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Abstract 

The  objective  of  this  study  was  to  determine  how  and  to 
what  degree  Air  Force  program  managers  evaluated  the  risks 
associated  with  the  various  program  decisions  in  which  risk 
assessments  are  mandated  by  federal  and  defense  regulations 
or  directives . 

This  preliminary  study  took  responses  from  a  series  of 
interviews  with  program  managers  to  assess  the  program 
managers'  knowledge  of  risk  and  risk  assessments,  how  they 
usually  performed  risk  assessments,  why  they  performed  risk 
assessments,  and  if  any  relationships  existed  between 
individual  backgrounds  and  risk  taking  behaviors. 

The  research  found  that  although  the  program  managers 
in  this  study  were  all  involved  with  risk  assessments  and 
analyses,  they  did  not  appear  to  be  properly  educated  in  the 
areas  of  risk  and  risk  assessment.  The  managers  were, 
however,  interested  in  learning  more  about  the  concept  of 
risk  and  strongly  suggested  the  addition  of  formal 
coursework  in  the  area  of  risk  management. 

This  study  also  showed  that  differences  are  likely  to 
exist  between  program  managers  possessing  technical  degrees 
(engineering,  basic  sciences,  etc.)  and  those  not  possessing 
technical  degrees  with  respect  to  how  cost  and  technical 
risk  are  evaluated  in  various  program  phases.  To  get  a  more 

V  i  i 


in-depth  understanding  of  the  relationship  of  risk  to  Air 


Force  weap''n  systems  acquisition,  it  is  important  to 
understand  why  these  differences  exist. 

Consequently,  it  is  recommended  that  further  research 
be  carried  out  to  more  precisely  measure  this  difference  in 
risk  taking  tendencies. 


V 1  i  1 


A  STUDY  OF  THE  ROLE  OF  RISK 
IN  THE  MANAGEMENT  OF 
AIR  FORCE  ACQUISITION  PROGRAMS 


I.  Introduction 

The  defense  acquisition  business  is  an  inherently  risky 

business  for  both  the  military  and  the  civilian  contractors 

who  produce  the  defense  weapon  systems.  Threats  are 

sometimes  difficult  to  [redict,  and  methods  for  countering 

these  threats  are  often  state-of-the-art  or  beyond.  Some  of 

these  defense  acquisition  programs  are 

designed  to  achieve  performance  levels  never 
before  realized,  using  many  components  and  some 
materials  never  before  used  in  military 
applications.  (16:10) 

As  a  consequence  of  working  in  this  risky  environment,  the 
acquisition  manager  must  make  decisions  considering  the 
"unpredictability  of  technical  performance,  development 
time,  and  cost"  (16;10'>. 

In  an  article  outlining  what  makes  a  good  project 
manager,  Thornberry  and  Weintraub  state  that  one  of  the  four 
primary  work  skills  of  a  project  manager  is  decision  making 
(33:75) . 

Project  Managers  were  good  decision  makers  and 
risk  takers.  They  had  to  gather  data,  refine  the 
problem,  deduce  causation,  develop  alternatives 
and  analyze  the  costs/benefits  of  each 
alternative  (33:75)  (emphasis  added) 

It  is  well  known  that  Air  Force  acquisition  program 

managers  take  risks  in  almost  every  decision  they  make; 


there  are  risks  associated  with  such  program  issues  as  cost 


schedule,  technical  performance,  supportability , 
programmatics,  etc.  Decisions  must  be  made  to  aHrlress  these 
risk  related  issues. 


Air  Force  Risk  Decision  Areas 

A  prime  example  of  an  acquisition  program  manager's 

decisions  involving  risk  is  the  set  of  decisions  associated 

with  contractor  source  selection.  The  Air  Force  regulation 

outlining  the  procedures  and  policies  involved  in  the  source 

selection  process,  AFR  70-15,  specifically  calls  for  a  risk 

assessment  as  part  of  the  proposal  evaluation  and  source 

selection  decision  (7:16). 

The  risks  which  must  be  assessed  are  those 
associated  with  cost,  schedule,  and  performance  or 
technical  aspects  of  the  program.  Risks  may  be 
inherent  in  a  program  by  virtue  of  the  program 
objectives  relative  to  the  state  of  the  art. 

Risks  may  occur  as  a  result  of  a  particular 
technical  approach,  manufacturing  plan,  the 
selection  of  certain  materials,  processes, 
equipment,  etc.,  or  as  a  result  of  the  cost, 
schedule  and  economic  impacts  associated  with 
these  approaches.  (7:17) 

The  program  office  is  responsible  for  preparing  a 
preliminary  risk  assessment  for  the  Source  Selection 
Evaluation  Board  prior  to  the  contractor's  proposal 
submissions.  This  is  done  so  that  the  board  can  compare  the 
contractors'  critical  program  risk  areas  with  those 
identified  by  the  government  (7:16). 

Department  of  Defense  Directive  5000.1,  Ha.ior  and  Non- 
Ha.ior  Defense  Acquisition  Programs,  outlines  the  policies 


2 


and  procedures  for  the  acquisition  of  defense  systems,  and 

risk  is  specifically  addressed  in  the  directive  on  two 

occasions.  First,  risk  is  called  out  as  a  decision  factor 

when  determining  whether  or  not  a  program  is  to  be 

designated  a  major  program.  A  program  that  is  determined  to 

have  a  high  degree  of  development  risk  can  be  designated  a 

major  program,  and,  hence,  require  Secretary  of  Defense  or 

Secretary  of  the  Air  Force  milestone  review  and  approval 

(9:2).  The  second  area  in  which  risk  is  addressed  by  DoD 

Directive  5000.1  is  in  the  development  of  a  program's 

acquisition  strategy.  The  directive  addresses  risk  in 

acquisition  strategy  with  the  following: 

Commensurate  with  risk,  such  approaches  as 
developing  separate  alternatives  in  high-risk 
areas;  using  early  funding  to  design-in 
reliability  and  support  characteristics ;  reducing 
lead  time  through  concurrency;  using  competitive 
prototyping  of  critical  components;  combining 
acquisition  phases  and  making  use  of  evolutionary 
acqu is  it  ion'  procedures ;  and  combining 
developmental  and  operational  test  and  evaluation 
shall  be  considered  and  adopted  when  appropriate. 

(9:5^ 

The  program  manager  must  make  decisions  on  whether  or 
not  to  incorporate  any  of  the  above  strategies  to  reduce  the 
technological,  schedule  and  cost  risks  of  a  program  (9:5). 

Related  to  the  acquisition  strategy  is  the  acquisition 
plan.  Part  7  of  the  Federal  Acquisition  Regulation  outlines 
the  necessary  components  of  the  acquisition  plan,  and  part 
of  Lhe  plan  is  a  discussion  of  risk  (24:300).  The 
acquisition  plan  for  a  major  system  must  do  the  following: 
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Discuss  technical,  cost,  and  schedule  risks  and 
describe  what  efforts  are  planned  or  underway  to 
reduce  risk  and  the  consequences  of  failure  to 
achieve  goals.  If  concurrency  of  development  and 
production  is  planned,  discuss  its  effects  on  cost 
and  schedule  risks.  (24:300) 

The  above  statement  highlights  the  fact  that  all 
aspects  of  risk  (cost,  schedule,  and  technical  performance) 
must  be  analyzed  together  because  they  may  have  effects  on 
each  other.  In  other  words,  if  technical  risk  is  reduced 
through  concurrency  of  development,  will  that  drive  up  cost 
or  lengthen  the  schedule?  This  planning  phase  of  the 
acquisition  cycle  is  where  most  of  the  risk  analysis  and 
related  decisions  are  made.  The  program  manager  must  answer 
questions  like, 

Is  the  system  mature  enough  to  impose 
specifications  and  standards  without  losing  a 
better  solution?  Have  I  enough  information  to 
streamline  testing  requirements?  Are  risks  low 
enough  to  impose  concurrency  of  test  and 
evaluation?  (29:37) 

Another  area  of  the  acquisition  plan  in  which  the 
program  manager  must  make  a  decision  involving  risk  is  in 
the  selection  of  a  contract  type.  The  program  manager  can 
choose  to  have  a  firm-fixed-price  type  of  contract  in  which 
the  contractor  assumes  all  financial  risks.  The  program 
manager  can  go  to  the  other  end  of  the  spectrum  and  choose 
to  have  a  cost-plus-fixed-fee  type  contract  in  which  the 
government  assumes  all  financial  risks,  or  he  or  she  can 
choose  a  shared  risk  type  of  incentivized  contract  somewhere 
between  the  two  extremes  (34).  This  decision  is  normally 
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based  on  the  level  of  definition  of  the  program.  If  the 
program  requirements  are  well  established  with  a  package  of 
specifications,  drawings,  and  other  data  which  dictate  what 
is  to  be  produced,  a  firm-fixed-price  production  type 
contract  is  used.  Conversely,  if  requirements  are  not  well 
defined,  a  cost  type  development  contract  is  normally  used 
(17:61). 

Test  and  evaluation  (T&E)  is  one  other  major  area  in 
which  the  acquisition  manager  addresses  program  risk.  One 
of  the  four  primary  purposes  of  a  T&E  program  is  "to 
identify,  assess,  and  reduce  the  acquisition  risks"  (8:2). 
The  acquisition  manager  addresses  the  risk  aspects  of  the 
program  when  developing  the  program  Test  and  Evaluation 
Master  Plan  (8:2). 

The  Air  Force  acquisition  program  manager  makes  program 
decisions  based  on  an  organizational  goal  of  weapon  system 
deployment  and  an  overall  goal  of  maintaining  "an  effective 
and  ready  fighting  force"  (4:14).  Above  are  just  a  few 
examples  of  where  a  program  manager  is  specifically  directed 
to  address  risk  in  the  acquisition  cycle.  The  program 
manager  must  weigh  alternatives  against  each  other  when 
making  program  decisions;  the  level  of  risk  associated  with 
each  alternative  is  a  major  factor  in  this  weighing  process. 

Role  cf  the  Program  Manager 

J.  Ronald  Fox  and  James  L.  Field  discuss  the  role  of 
the  program  manager  with  the  following: 
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As  the  key  figures  in  the  acquisition  process, 
program  managers  must  oversee  the  efforts  of  their 
military  services  to  acquire,  deploy  or  operate, 
and  support  major  weapon  systems  of  proven 
capability  within  approved  schedules  and  budgets. 

To  fulfill  these  objectives,  they  are  expected  to: 

1.  establish  firm  and  realistic  system  and 
equipment  specifications; 

2.  define  organizational  relationships  and 
responsibilities; 

3.  identify  high-risk  areas;  (emphasis  added) 

4.  select  the  best  technical  approaches; 

5.  explore  schedule,  cost,  and  technical 
performance  trade-off  decisions; 

6.  establish  firm  and  realistic  schedules  and 
cost  estimates; 

7.  formulate  realistic  logistics  support  and 
operational  concepts;  and 

8.  lay  the  groundwork  for  contracting  for  the 
program.  (16:21) 

Risk  is  specifically  addressed  by  Fox  and  Field  in  item 
3  above,  but  in  order  to  analyze  specifications,  evaluate 
technical  approaches,  make  trade-offs,  and  make  schedule  and 
budget  decisions,  the  program  manager  must  know  the  elements 
of  risk  associated  with  each  and  make  that  a  factor  in  his 
or  her  decision  process  (16:21). 

DoD  Directive  4245.7,  the  defense  guide  for 
transitioning  defense  programs  from  development  to 
production,  requires  that  "elements  of  program  risk  shall  be 
identified  and  assessed  throughout  the  acquisition  cycle" 
(10:2).  A  manual  accompanying  this  directive  analyzes  the 
acquisition  process  and  the  possible  elements  of  risk  at  all 
phases  of  development.  The  manual  is  a  useful  guide  for 
identifying  risks,  but  it  provides  no  means  of  assessing  or 
quantifying  the  risks.  The  manual  divides  the  acquisition 
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process  into  phases  of  design,  test,  and  production,  and 
provides  templates  of  these  phases.  These  templates  address 
activities  under  each  phase.  For  example,  under  the  test 
template,  eight  sub-categories  of  testing  are  identified, 
and  possible  risky  areas  are  identified  under  each  (10:4-2). 

Even  though  methods  for  analyzing  potential  risk  areas 
are  in  existence,  there  are  no  standardized  methods  for 
performing  risk  assessments  within  the  Department  of 
Defense.  The  method  used  is  left  to  the  discretion  of  the 
individual  program  manager.  Also,  there  is  no  standard  way 
of  quantifying  risk.  Air  Force  Regulation  70-15  defines 
risk  as  being  "high,"  "moderate,"  or  "low,"  but  this  is  not 
considered  a  standard  to  be  used  for  all  types  of  risk 
assessments  (7:16). 

Problem  Statement 

Acquisition  managers  are  required  by  regulation  to 
manage  risk  and  to  perform  risk  assessments  and  analyses 
when  making  decisions  for  various  aspects  of  their  programs. 
However,  clear  and  explicit  guidance  as  to  how  to  perform 
these  risk-related  tasks  is  not  provided.  Without  knowledge 
as  to  "how"  to  manage  and  assess  risk,  it  is  likely  that 
inconsistent  management  of  risk  will  occur. 

Qbje.c..tiYe 

The  overall  objective  of  this  research  is  to  determine 
how  and  to  what  degree  Air  Force  program  managers  evaluate 
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the  risk  associated  with  the  various  program  decisions 
listed  below: 

1.  Contractor  source  selection. 

2.  Preparing  the  Acquisition  Strategy. 

3.  Preparing  an  Acquisition  Plan  risk  assessment  for 
the  Procuring  Contracting  Officer  (PCO). 

4.  Determining  the  type  of  contract  to  be  used. 

5.  Determining  Test  and  Evaluation  objectives. 

6.  Other  program  specific  areas. 

Investigative  Questions 

To  meet  the  overall  objective,  the  following 
investigative  questions  must  be  answered. 

1.  How  do  program  managers  view  the  concept  of  risk? 

2.  How  much  risk  do  they  feel  is  present  in  each  of 
the  program  management  decision  areas  outlined  above,  and 
what  kinds  of  risks  are  they? 

3.  What  means  do  program  managers  use  to  evaluate 
(formally  or  informally,  quantitatively  or  qualitatively) 
risk  in  each  area? 

4.  What  circumstances  lead  them  to  evaluate  risk  in 
these  ways? 

5.  Do  correlations  exist  between  risk  taking 
tendencies  and  the  following  independent  variables: 


a . 

Academic 

Degree? 

c  . 

Age? 

b. 

Types  of 

Work  Experiences? 

d  . 

Marital  Status? 
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In  question  5  above,  the  variable  "academic  degree" 
referred  to  whether  or  not  the  program  manager  held  a 
technical  degree.  A  technical  degree  was  defined  to  be  any 
scientifically  based  degree  such  as  engineering  or 
chemistry . 

The  variable  "types  of  work  experiences"  actually 
referred  to  two  variables  in  the  correlation  analysis.  The 
first  of  these  two  was  whether  or  not  the  program  manager 
had  any  operational  experience.  Operational  experience  was 
defined  to  be  any  experience  outside  of  the  Air  Force 
Systems  Command  systems  acquisition  environment  which  dealt 
with  the  day-to-day  missions  of  the  Air  Force.  Examples  of 
operational  experience  included  such  jobs  as  pilot, 
navigator,  missile  officer,  maintenance  officer,  etc.  The 
second  of  the  work  experience  variables  was  whether  or  not 
the  program  manager  had  worked  at  a  base  other  than  Wright- 
Patterson  Air  Force  Base.  The  other  two  variables  used  in 
the  correlation  analysis,  "age"  and  "marital  status,"  were 
self  explanatory. 

Rationale 

There  is  a  basic  problem  with  making  decisions 

involving  risk  in  the  systems  acquisition  environment. 

Risk  management  for  the  DoD  system  acquisition 
process  is  not  an  exact  science.  There  is  no 
common  or  standard  DoD  sanctioned  practice, 
technique,  or  procedure  for  identifying,  assessing 
and  managing  risk.  Furthermore,  there  is  no 
universally  accepted  definition  of  risk.  (20:192) 
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Because  there  is  no  standardized,  quantitative  or 
qualitative  way  of  measuring  program  risk  in  the  acquisition 
environment,  the  risk  assessments  and  analyses  associated 
with  Air  Force  programs  are  highly  subjective  and  based 
mainly  on  the  individual  risk  taking  tendencies  of  the 
program  manager.  Other  research  has  shown  that  people  do 
not  evaluate  risks  in  the  same  manner,  leading  to  a  variety 
of  decisions  when  faced  with  risk  and  uncertainty  (12:3). 
Understanding  the  risk  taking  tendencies  of  program  managers 
can  serve  as  a  starting  point  for  developing  more  objective 
criteria  for  evaluating  and  assessing  the  risks  associated 
with  the  many  decisions  common  to  program  management.  With 
the  recent  strong  interest  in  risk  assessment  in  the 
acquisition  community,  developing  a  greater  understanding  of 
risk  in  that  environment  should  be  useful  to  the  program 
manager . 

Summary 

The  purpose  of  this  introductory  chapter  was  to 
acquaint  the  reader  with  some  of  the  issues  involving  risk 
and  the  Air  Force  acquisition  program  manager.  In  addition, 
it  presented  the  problem  statement,  the  objective  of  this 
research,  and  the  rationale  for  this  research.  Chapter  II 
is  a  review  of  the  literature  concerning  risk,  uncertainty, 
and  how  they  are  addressed  in  the  Air  Force  acquisition 
process.  Chapter  III  is  a  discussion  of  the  methodology 
used  to  carry  out  this  research.  Chapter  IV  is  a  summary  of 
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the  results  of  the  research,  and  the  final  chapter  contains 
conclusions  and  recommendations  for  any  possible  solutions 
and  for  further  studies  in  the  area  of  risk  and  acquisition 
management . 
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The  concept  of  risk  has  been  widely  studied  and 
documented,  and  opinions  vary  widely  as  to  what  exactly 
constitutes  risk.  These  semantic  differences  are  addressed 
in  this  review  of  the  literature.  First,  risk  is  defined  in 
a  general  sense.  This  includes  a  discussion  on  how  risk 
relates  with  utility  theory  and  what  sets  risk  apart  from 
"uncertainty . " 

Following  the  general  discussion  of  risk  is  a  more 
specific  discussion  on  how  managers  view  risk  and  how  that 
risk  can  be  measured.  Finally,  the  literature  review 
focuses  on  risk  in  the  realm  of  the  Department  of  Defense 
and  the  Air  Force  systems  acquisition  process.  This 
includes  a  review  of  several  models,  techniques,  and  other 
tools  available  to  the  program  manager  for  managing  risk. 

It  should  be  noted  that  although  these  tools  exist,  no 
guidance  was  found  to  mandate  their  use. 

Definition  of  Risk 

Webster's  Contemporary  American  Dictionary  defines  risk 
as  "the  possibility  of  suffering  harm  or  loss;  danger" 
(35:608).  MacCrimmon  and  Wehrung  break  this  definition  down 
further  into  three  parts. 

First,  it  is  necessary  that  there  be  a  potential 

Joss  of  some  amount.  Second,  there  must  be  a 

chance  of  loss.  A  sure  loss  is  not  a  risk. 

Third,  the  notion  "to  expose"  means  that  a 
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decision  maker  can  take  actions  that  can  increase 
the  magnitude  or  chance  of  loss.  (26:9) 

MacCrimmon  and  tfehrung  also  point  out  two  types  of  losses, 

the  first  being  something  that  will  make  one  "worse  off" 

than  he  or  she  is  presently,  the  second  being  something  that 

is  not  as  good  as  something  else  that  could  have  been 

obtained  (26:10). 

Fischoff  and  others,  distinguish  a  risk  problem 
decision  from  an  ordinary  decision  in  that  risk  problems 
include  some  threat  to  life  or  health  among  its  consequences 
(15:2).  For  the  purposes  of  this  research,  that  threat  to 
life  or  health  is  expanded  to  a  threat  involving  any  type  of 
loss  . 

Grose  states  that  risk  has  a  large  number  of 
components . 

A  totally  inclusive  definition  of  risk  includes 
hazards,  dangers,  potential  for  loss,  the  degree 
or  probability  of  a  specific  exposure  for  loss  as 
well  as  the  liability  to  injury,  damage,  loss,  or 
pain.  It  encompasses  jeopardy  or  the  exposure  to 
extreme  danger  for  any  situation.  Events  with 
both  chance  and  voluntary  provocation  are 
included.  Loss  potential  due  to  risk  also 
embraces  rational  behavior,  irrational  behavior, 
natural  phenomena,  and  any  other  potential  for 
realizing  unwanted,  negative  consequences  of  any 
event .  (18:24) 

This  definition  is  simplified  by  Grose  to  be  "the  likelihood 
of  injury,  harm,  damage  or  loss  multiplied  by  its  potential 
magnitude"  (18:25).  The  term  "likelihood”  implies  some  sort 
of  probability  of  occurrence.  According  tc  Peter  Sprent, 
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any  study  of  risk  or  risk  management  must  involve  the  use  of 

statistical  methods  to  assess  these  probabilities. 

It  calls  for  an  understanding  of  how  to  collect 
and  interpret  relevant  factual  evidence.  Equally 
importantly,  it  demands  an  appreciation  of  the 
psychology  of  human  reaction  to  uncertainty  and 
risk;  this  is  linked  to  both  how  we  perceive 
danger  and  to  our  ability  to  interpret  correctly 
all  relevant  factual  information. 

Byrne  and  Cadman  also  emphasize  the  importance  of 

gathering  necessary  and  relevant  information  when  assessing 

risks  and  making  decisions.  They  underscore  the  need  for 

sifting  through  given  information: 

In  one  way  or  another,  objectively  or 
subjectively,  the  developer  or  decision-maker  has 
to  sort  out  or  analyze  the  information  that  he  is 
given  and  then  make  the  best  decision  he  can. 

Because  these  decisions  have  to  be  made  very 
largely  on  the  basis  of  estimates  or  expectations 
of  the  future,  the  developer  has  to  contend  with 
uncertainty.  (3:24) 

The  Defense  Systems  Management  College  (DSMC)  guide  to 
risk  management  defines  risk  as  "the  probability  of  an 
undesirable  event  occurring  and  the  significance  of  the 
consequence  of  the  occurrence"  (6:3-1).  This  DSMC 
definition  is  similar  to  Grose's  in  that  both  emphasize  the 
magnitude  or  significance  of  the  consequence.  The 
relationship  between  the  probability  of  occurrence  and  the 
severity  ot  the  consequence  and  how  they  combine  to  produce 
a  level  of  risk  is  illustrated  in  Figure  1. 
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SEVERITY  OF  CONSEQUENCE 


Figure  1.  Concept  of  Risk 

(Reprinted  from  6;3-2) 
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Figure  1  shows  four  quadrants.  Three  of  the  four 
quadrants  contain  a  known  level  of  risk.  The  low 
probability/low  severity  quadrant  is  an  area  of  low  risk. 

The  high  probability/low  severity  quadrant  is  generally  low 
risk,  and  the  high  probability/high  severity  quadrant  is  an 
area  of  high  risk. 

The  low  probability/high  severity  quadrant  is  the  area 
in  which  program  manager  individual  interpretation  plays  a 
major  role  in  determining  the  risk  level.  If  the 
consequence  is  extremely  severe,  the  risk  may  be  high  even 
with  an  extremely  low  probability  of  occurrence  (6:3-2).  An 
example  of  this  type  of  high  risk  would  be  the  acquisition 
of  pressure  suits  for  the  space  shuttle  astronauts.  The 
pressure  seals  in  the  suits  must  be  as  close  to  100% 
reliable  as  possible.  Any  failure  is  unacceptable,  and, 
hence,  the  item  could  be  considered  high  risk. 

The  DSMC  guide  also  notes  that  overall  program  risk  is 
dependent  upon  the  risks  associated  with  each  component  of 
the  system.  A  large  number  of  moderate  risk  components  can 
result  in  a  program  with  an  overall  high  risk  rating.  Also, 
a  program  with  only  a  small  number  of  high  risk  items  may 
result  in  a  program  with  an  overall  low  risk  rating.  The 
program  manager  must  use  some  model  or  tool  to  make  this 
overall  program  assessment.  Some  of  these  models  and  tools 
are  discussed  later  in  this  literature  ' ’view  (6:3-2). 
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Byrne  and  Cadman  believe  that  an  individual's 
perspective  of  risk  and  risk  preference  is  best  described 
through  the  use  of  a  utility  curve.  An  individual  can  be 
shown  to  be  a  risk-taker,  a  risk-seeker,  or  a  risk-neutral 
person  depending  on  the  shape  of  the  curve  (15:54). 
Management  scientists  use  utility  theory  to  determine  these 
risk  taking  tendencies.  Three  concepts  used  to  determine 
these  tendencies  are.  Expected  Value,  Certainty  Equivalent, 
and  Risk  Premium  (36:582). 

An  individual's  risk  taking  tendencies  can  best  be 
understood  by  considering  the  following  example. 

Suppose  an  individual  is  given  a  choice  of  two  outcomes 
with  equal  probability  (a  50-50  chance).  The  first  is  that 
he  or  she  gains  $30,000;  the  second  is  that  he  or  she  loses 
$10,000.  The  expected  value  is  equal  to  the  sum  of  each 
outcome  multiplied  by  its  respective  probability  of 
occurrence.  In  this  case: 

Expected  Value  =  ( 30 , 000 )*( . 5 )  +  ( -10, 000)*( . 5)  =  $10,000 
The  certainty  equivalent  is  the  value  at  which  the 
decision  maker  is  indifferent  between  the  above  choice  of 
outcomes  and  receiving  a  certain  outcome.  For  example,  he 
or  she  may  be  indifferent  between  losing  $3400  for  sure  and 
deciding  between  gaining  $30,000  or  losing  $10,000.  In  this 
case  : 

Certainty  Equivalent  =  -$3400 
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Finally,  risk  premiun  is  the  difference  between  the 
expected  value  and  the  certainty  equivalent.  In  this  case: 

Risk  Premium  =  10,000  -  (-3400)  =  $13,400 

This  individual  "values  (the  choice)  at  $13,400  less 
than  its  expected  value,  because  (he  or  she)  does  not  like 
the  large  degree  of  uncertainty  associated  with  the  reward 
yielded  by  (the  choice)"  (36:582).  This  positive  result 
implies  risk-averse  behavior.  A  risk  premium  equal  to  zero 
implies  risk-neutral  behavior,  and  a  negative  risk  premium 
implies  risk-taking  behavior  (36:582). 

A  study  by  Kahneman  and  Tversky  tested  utility  theory 
to  evaluate  people's  risk  taking  tendencies.  In  addition  to 
the  process  described  above,  they  studied  the  domain  in 
which  the  decisions  were  made.  The  domain  of  a  decision  was 
either  positive  or  negative.  Their  results  showed  that 
individuals  tended  to  be  risk  averse  in  the  positive  domain. 
A  gain  of  $3000  for  sure  was  preferred  to  a  gain  of  $4000 
with  an  80%  probability  (expected  value  of  3000  preferred 
over  expected  value  of  3200).  However,  these  same 
individuals  were  risk  seeking  in  the  negative  domain.  A 
loss  of  $4000  with  an  80%  probability  was  preferred  to  a 
loss  of  $3000  for  sure  (expected  value  of  -3200  preferred 
over  expected  value  of  -3000).  These  results  are 
inconsistent  with  pure  utility  theory  which  would  suggest 
that  the  domain  would  make  no  difference.  The  primary 
reason  for  this  difference,  they  suggest,  is  that  people 
tend  to  overweight  certainty  (23:269-275). 
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Byrne  and  Cadman  relate  risk  and  uncertainty  with  the 
following : 

...uncertainty  is  taken  to  be  anything  that  is  not 
known  about  the  outcome  of  a  venture  at  the  time 
when  the  decision  is  made.  In  contrast,  risk  is 
taken  to  be  the  measurement  of  a  loss,  identified 
as  a  possible  outcome  of  the  decision.  (15:10) 

The  relationship  of  risk  and  uncertainty  was  addressed 

by  Kahan  in  the  context  of  decision  making. 

Consequences  result  as  a  function  of  our  choice. 

If  the  decision  maker's  choice  uniquely  determines 
the  consequences,  then  we  speak  of  decision  making 
under  certainty.  If,  on  the  other  hand, 
consequences  occur,  given  a  choice,  with  a  known 
(or  estimable)  probability,  then  we  speak  of 
decision  making  under  risk.  Finally,  if  we  don't 
(or  can't)  know  the  probaoilit ies ,  then  we  speak 
of  decision  making  under  uncertainty.  (22:1) 

(emphasis  added) 

The  concept  of  uncertainty  can  be  thought  of  as  a 
continuum  ranging  from  complete  certainty  to  complete 
uncertainty.  According  to  Adams  and  Martin,  the  level  of 
uncertainty  is  inversely  related  to  the  amount  of 
information  available  to  the  decision  maker.  With  total 
information,  there  is  no  uncertainty  present,  and  a  decision 
is  made  with  a  known  outcome  (no  risk).  Risk  enters  the 
uncertainty  continuum  when  the  decision  maker  has  partial 
information  about  a  negative  outcome.  This  partial 
information  is  the  likelihood  or  probability  of  an  outcome. 
The  "risk"  is  the  possible  undesired  outcome  and  its 
associated  probability  of  occurrence.  As  the  level  of 
uncertainty  increases  along  the  continuum,  a  region  of 
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complete  uncertainty  is  encountered  where  no  information  is 
available  to  the  decision  maker,  and  all  decisions  are  made 
with  no  known  or  estimable  outcome  probabilities  (1:36). 

Management's  View  of  Risk 

The  literature  suggests  that  managers  view  risk  with  a 
less  precise  view  than  those  of  the  definitions  discussed 
above.  March  and  Shapira  suggest  three  differences.  First, 
most  managers  do  not  feel  that  the  positive  consequences  of 
a  decision  are  an  important  aspect  of  risk.  In  other  words, 
a  decision  which  involves  only  positive  outcomes  with  their 
respective  probabilities  does  not  involve  risk.  Risk  is 
strictly  associated  with  negative  outcomes.  Second,  many 
managers  see  uncertainty  as  a  factor  in  risk.  In  other 
words,  risk  is  not  uniquely  made  up  of  probablistic 
outcomes.  Thirdly,  "managers  show  little  desire  to  reduce 
risk  to  a  single  quantifiable  construct'  (27:1407).  Many 
managers  resist  using  available  tools  to  quantify  the 
factors  which  enter  into  a  risk  decision  or  assessment. 
Managers  tend  to  stress  the  importance  of  such  things  as 
personal  judgment  and  expertise  as  part  of  their  risk 
calculations.  Managers  need  to  have  a  "gut  feel"  for  the 
riskiness  of  a  situation  rather  than  simply  rely  on  computer 
generated  data  based  on  some  specific  set  of  algorithms 
(27:1407). 
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Why  PQ-hanagers  Take  Risk? 

March  and  Shapira  point  out  three  reasons  managers  take 
risks. 

1.  Risk  is  essential  to  success  in  decision 

making . 

2  Risk  taking  is  an  essential  component  of  the 

managerial  role. 

3.  Managers  recognize  emotional  pleasures  and 

pains.  (27:1409) 

Their  research  found  that  managers  feel  that  in  order 
to  be  successful  in  business,  a  manager  must  take  risks  when 
making  decisions,  and  this  relates  to  his  managerial  role. 
They  also  suggest  that  successes  are  more  satisfying  when  a 
chance  is  taken,  and  these  chances  cannot  be  taken  without 
risk.  The  thrill  of  making  a  decision  under  risk  keeps  the 
job  challenging  and  interesting  (27:1409). 

Risk  Measurement 

To  measure  risk  propensity  among  managers  in  their 
studies,  MacCrimmon  and  Wehrung  used  two  types  of  risk 
measures : 

1,  Primary  Measures.  These  consisted  of  standardized 
situations  and  allowed  for  meaningful  comparison.  Examples 
of  these  included  investment  gambles  for  both  the  business 
and  the  individual  and  pre-defined  risky  office  situations 
such  as  a  lawsuit,  a  customer  threat,  or  a  union  dispute. 
These  measures  permitted  analysis  of  general  risk  taking 
tendencies  (26:183-185,207). 
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2.  Secondary  Measures.  These  consisted  of  Attitudes 
and  Natural  Situations  which  were  observed.  These  were  used 
to  evaluate  whether  or  not  any  individual  acted  in  a  real- 
life  situation  as  he  or  she  would  in  a  standardized 
situation.  From  these  observations,  inferences  were  made 
(26:183-185,207). 

These  authors  and  researchers  have  shed  light  upon  some 
general  aspects  of  risk  and  uncertainty.  Their  different 
perspectives  indicate  the  great  variability  associated  with 
how  to  manage  risk  in  general.  The  next  section  describes 
how  these  general  risk  issues  have  been  addressed  in  the 
acquisition  of  Department  of  Defense  and  Air  Force  weapon 
systems . 

Risk- ia  Air  force  Systems  Acguisitioii 

A  network  analysis  manual  sums  up  the  origin  of  risk 

analysis  in  defense  acquisition  programs  as  follows: 

With  the  advent  of  the  large  cost  overruns  and 
schedule  slippages  experienced  on  many  major 
development  projects  in  the  defense  sector  of  the 
U.S.  economy  in  the  late  sixties  and  early 
seventies,  military  managers  realized  a  need  for 
RISK  ANALYSIS.  (28:2) 

Initially,  the  Army  was  the  lead  agency  for  developing 
methods  for  risk  analysis.  One  of  these  methods  was  a 
computerized  networking  technique  called  the  Venture 
Evaluation  and  Review  Technique  (VERT).  What  set  VERT  apart 
from  previous  networking  techniques  was  the  fact  that  it 
could  handle  extremely  complex  decision  problems  in  terms  of 
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cost  and/or  schedule  and/or  performance  rather  than  simply 
time  alone.  Hence,  a  rink  assessment  could  be  performed 
incorporating  all  elements  of  program  risk  (28:1-3).  The 
Air  Force  followed  suit  by  establishing  their  own 
computerized  networking  program  for  analyzing  risk.  This 
program,  maintained  at  the  Armament  Division,  Eglin  AFB ,  FL , 
is  called  "the  RISK  model"  (5:VI-5). 

Because  of  the  increasing  level  of  technological 
complexity  of  defense  programs  and  the  associated  risks 
involved,  Deputy  Secretary  of  Defense  Carlucci  initiated  the 
Acquisition  Improvement  Program  (AIP)  in  1981.  One  of  the 
initiatives  was  to  increase  the  awareness  of  technological 
risk  and  the  costs  involved  in  performing  inadequate  risk 
assessments.  This  initiative  was  called  "Budgeting  for 
Technological  Risk"  and  called  for  program  managers  to  allow 
for  an  increased  management  reserve  budget  to  account  for 
possible  cost  growth  due  to  technological  complexity 
(2:375)  . 

What  resulted  from  the  Office  of  the  Secretary  of 
Defense  interest  in  risk  assessment  was  the  development  of 
cost/risk  assessment  concepts  based  on  the  Army's  Total  Risk 
Assessing  Cost  Estimate  (TRACE)  methodology.  The  TRACE 
method  used  a  risk  factor  approach  in  which  components  of  a 
program  were  assigned  a  factor  based  on  an  estimate  of  their 
relative  risks.  These  factors  were  applied  to  the  estimated 
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cost  and  resulted  in  a  revised  cost  estimate  known  as  the 
TRACE  (5  .'Appendix  1-2). 

The  various  methods  for  performing  risk  assessments  are 
outlined  in  the  DSMC  guide  for  risk  management.  Methods 
that  can  be  used  include  the  following: 

1.  Expert  Interviews, 

This  involves  questioning  experts  in  certain  areas  about  the 
risk  aspects  of  their  specialties  (6:5-4). 

2.  Analogy  Comparison/Lessons  Learned  Studies. 

This  involves  comparing  programs  with  similar  past  programs 

and  their  associated  risks  (6:5-7).  The  DSMC  guide  on  risk 

management  states  the  following: 

...key  insights  can  be  gained  concerning  the 
various  aspects  of  a  current  program's  risk,  by 
examining  the  successes,  failures,  problems,  and 
solutions  of  similar  existing  or  past  programs. 

(6:5-7) 

3.  Plan  Evaluation. 

This  involves  addressing  risk  in  all  p 
relating  to  planning  and  ensuring  that 
inconsistencies  or  contradictions  (6:5 
these  program  documents  are  the  follow 

•  Program  Management  Plan  (PMP) 

•  Systems  Engineering  Management 

•  Acquisition  Plan  (AP) 

•  Test  and  Evaluation  Master  Plan  (TEMP) 

•  Manufacturing  Plan  (HP) 


rogram 

documents 

there 

are  no 

-12). 

Examples  of 

ing : 

Plan  (SEMP) 
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Integrated  Logistics  Support  Plan  (ILSP) 


•  Work  Breakdown  Structure  (WBS)  Index  and 
Dictionary 

•  Specifications-  and  the  Specification  Tree 

•  Statements  of  Work 

•  Other  "Baseline"  Documents  (6 ; 5-11 , 5-12  ) 


4.  Transition  Templates. 

This  involves  planning  for  risk  using  the  templates  found  in 
DoDD  4245. 7-M  (6:5-19). 

Each  template  examines  an  area  of  risk  and  then 
describes  methods  for  avoiding  or  reducing  that 
risk.  Much  of  the  description  of  the  risk  and  the 
solution  is  based  on  lessons  learned  from  other 
programs.  (6:5-19) 

In  addition  to  the  qualitatively  based  methods 
described  above,  some  mathematical  or  quantitative  tools  for 
performing  risk  analysis  are  also  described  in  the  DSMC 
guide.  These  include  the  following: 

1.  Decision  Analysis. 

This  involves  choosing  among  several  decision  alternatives 
using  probabilities,  states  of  the  world,  and  expected 
monetary  values  as  inputs  and  decision  criteria  (6:5-21,5- 
22)  . 


2.  Cost  Estimating  Relationships. 

This  involves  using 

an  equation  to  determine  an  appropriate  management 
reserve  or  risk  funds  budget.  When  using  this 
method,  the  management  reserve  funds  represent  the 
amount  of  funding,  over  and  above  that  determined 
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by  cost  analysis  alone,  required  for  work 
associated  with  unanticipated  risks.  (6:5-25) 

The  equation  is  based  on  correlations  of  costs  with  certain 

performance  characteristics  (6:5-25). 

3.  Network  Analysis. 

This  involves  establishing  a  critical  program  path  and 
identifying  potential  areas  of  schedule  risk  (6:5-29). 

4.  Life  Cycle  Cost  Analysis. 

This  involves  "a  series  of  equations  which  compute  program 
costs  based  on  product  and  program  information"  (6:5-34). 

5.  Cost  Risk/WBS  Simulation  Model. 

This  involves  the  use  of  computers  to  simulate  what  will 
happen  to  program  costs  given  the  probablistic  outcomes  of 
each  Work  Breakdown  Structure  cost  element  (6:5-37). 

As  previously  stated,  the  objective  of  this  research  is 
to  determine  how  and  to  what  degree  Air  Force  program 
managers  evaluate  the  risks  associated  with  certain  program 
decisions.  The  above  review  of  the  literature  suggests  that 
the  tools  and  methods  for  risk  assessment  are  available  to 
the  program  manager.  The  methodology  in  the  following 
chapter  was  designed  to  determine  if,  how,  and  to  what 
extent  Air  Force  program  managers  at  the  Aeronautical 
Systems  Division  (ASD),  Wright-Patterson  AFB ,  Ohio  use  these 
tools  and  methods  in  managing  their  programs. 
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This  chapter  describes  the  methods  used  to  determine 
how  and  to  what  degree  Air  Force  acquisition  program 
managers  evaluate  and  assess  the  many  risks  associated  with 
their  programs.  Specifically,  this  research  focused  on  the 
areas  of  Air  Force  program  management  in  which  risk 
assessments,  analyses,  and  evaluations  are  mandated  by 
regulation  or  directive.  The  purpose  of  this  chapter  is  to 
present  the  research  population,  the  methods  of  data 
collection  and  reduction,  and  any  research  assumptions  or 
limitations . 

The  review  of  the  literature  concerning  risk  in  Air 
Force  program  management  revealed  several  areas  in  which 
risks  were  to  be  addressed  by  the  program  manager  throughout 
the  life  of  the  program.  For  the  purposes  of  this  research, 
the  term  "program  manager"  referred  to  any  person  in  a 
program  office  concerned  with  the  day-to-day  management 
aspects  of  the  program.  These  people  could  range  in  rank 
from  the  General  or  Colonel  acting  as  program  director  to 
the  Second  Lieutenant  in  charge  of  test  planning.  Since 
this  definition  of  a  program  manager  covered  such  a  wide 
spectrum  of  personnel,  the  population  in  this  study  was 


limited  to  mid-level  military  and  civilian  program  managers 
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(Captain  through  Lieutenant  Colonel  and  civilian  grade 
equivalents ) . 

Population  and  Sample  Description 

Population .  Because  of  time,  physical,  and  financial 

constraints  the  population  of  this  study  was  limited  to 

analyzing  the  risk  taking  views  of  mid-level  Air  Force 

acquisition  program  managers  at  the  Aeronautical  Systems 

Division  (ASD),  Wr ight-Patterson  AFB ,  OH.  Each  of  the 

interview  subjects  carried  a  primary  Air  Force  Specialty 

Code  (AFSC)  of  27XX,  Acquisition  Manager. 

Sample .  The  sample  of  interest  was  drawn  from  the 

population  of  mid-level  acquisition  managers  located  at  ASD. 

This  sample  represented  a  cross  section  of  program  managers 

from  both  major  and  non-major  systems  at  ASD.  A  major 

system  is  defined  by  DoD  Directive  5000.1  to  be  any 

program  designated  so  by  the  Secretary  of  Defense  or  any 

program  requiring  the  following: 

...total  expenditure  for  research,  development, 
test,  and  evaluation  of  more  than  $200  million 
(based  on  Fiscal  Year  1980  constant  dollars)  or  an 
eventual  total  expenditure  for  procurement  of  more 
than  $1  billion  (based  on  Fiscal  Year  1980 
constant  dollars).  (9:2) 

Emory  states  that  samples  can  be  taken  based  either  on 
probability  techniques  or  on  nonpi obabi 1 ity  techniques. 
Probability  sampling  employs  some  sort  of  random  sample 
selection  technique.  Examples  of  these  are  simple  random 
sampling,  stratified  sampling,  cluster  sampling,  systematic 
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sampling,  and  double  sampling.  Nonprobability  sampling,  in 
contrast,  uses  no  random  sampling  techniques.  What  this 
means  is  that  "each  member  does  not  have  a  known  nonzero 
chance  of  being  included"  (14:278).  Emory  suggests  that 
nonprobability  sampling  is  useful  in  studies  that  are 
"looking  only  for  a  feel  of  the  range  of  conditions" 
(14:279).  Emory  also  points  out  that  nonprobability 
techniques  are  advantageous  because  of  reduced  cost  and  time 
requirements.  Because  this  was  an  exploratory  study  lookiiig 
for  a  "feel"  of  the  Air  Force  program  manager's  view  of 
risk,  nonprobability  sampling  was  the  technique  used.  This 
was  an  acceptable  method  given  that  care  was  taken  to 
control  the  sampling  process.  Emory  states  "Carefully 
controlled  nonprobability  sampling  often  seems  to  give 
acceptable  results"  (14:279). 

The  method  of  nonprobability  sampling  used  in  this 
study  was  called  purposive  sampling.  A  purposive  sample  is 
one  which  draws  subjects  based  on  certain  pre-specif ied 
criteria.  The  form  of  purposive  sampling  most  appropriate 
for  this  study  was  judgment  sampling.  "Judgment  sampling 
occurs  when  one  handpicks  sample  members  to  conform  to  some 
criterion"  (14:280).  The  membt "s  chosen  for  this  study  were 
chosen  based  on  the  criterion  that  they  performed,  were 
involved  with,  or  were  familiar  with  some  sort  of  risk 
analysis  or  assessment  in  the  management  of  their  Air  Force 
acquisition  programs.  Emory  states,  "When  used  in  the  early 
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stages  of  an  exploratory  study,  a  judgment  sample  is  quite 
appropriate"  (14:280). 

The  nonprobability  sampling  technique  used  in  this 
study  produced  results  in  which  generalizations  could  be 
made  about  population  parameters. 

Data  Collection  Technique 

The  primary  method  for  data  collection  was  a  series  of 

25  personal  interviews  with  Air  Force  program  managers.  As 

stated  above,  the  source  was  a  representative  sample  of 

managers  from  the  Air  Force  Systems  Command  product  division 

located  at  Wright-Patterson  AFB .  Parten  states  some 

advantages  of  the  personal  interview; 

The  personal  interview  usually  yields  a  high 
percentage  of  returns,  for  most  people  are  willing 
to  cooperate. 

The  information  secured  is  likely  to  be  more 
correct  than  that  secured  by  other  techniques 
since  the  interviewer  can  clear  up  seemingly 
inaccurate  answers  by  explaining  the  questions  to 
the  informant. 

The  interviewer  may  catch  the  informant  off  guard 
and  thus  secure  more  spontaneT^us  reactions  than 
would  be  the  case  if  a  written  form  were  mailed 
out  for  the  informant  to  mull  over.  (30:79-80) 

A  weakness  in  this  third  point  above  is  that  little 

thought  may  be  put  into  the  response  to  a  question.  To 

overcome  this  weakness,  prior  to  all  interviews, 

interviewees  were  provided  with  a  summary  of  questions  to  be 

asked  during  the  interview  so  that  some  pre-thought  could  be 

accomplished  concerning  the  general  issues  of  the  interview. 
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Spontaneous  reactions  would  then  still  be  possible  when 
desirable,  but  thought  out  responses  could  also  be  obtained 
when  appropriate.  Interviews  were  conducted  to  determine 
each  individual  subject's  views  on  risk  and  uncertainty, 
their  risk  taking  tendencies  in  the  program  management 
arena,  and  their  methods  of  managing  and  assessing  risk  as 
program  managers . 

The  interview  guide  consisted  of  questions  concerning 
the  specific  Air  Force  risk  assessment  areas  outlined  in 
Chapter  I.  Those  were: 

1.  Contractor  Source  Selection. 

2.  Preparing  the  Acquisition  Strategy. 

3.  Preparing  an  Acquisition  Plan  risk  assessment  for 
the  Procuring  Contracting  Officer  (PCO). 

4.  Determining  the  type  of  contract  to  be  used. 

5.  Determining  Test  and  Evaluation  objectives. 

6.  Other  Program  Specific  Areas. 

The  guide  primarily  consisted  of  open-ended  questions. 
Labaw  discusses  the  two  types  of  open-ended  questions.  One 
type  of  open-ended  question  allows  the  respondent  to  freely 
answer  the  question  in  any  way  he  or  she  chooses.  The 
second  type  of  open-ended  question  contains  precoded  answer 
categories.  Both  of  these  types  of  questions  were  used  in 
the  construction  of  the  interview  guide.  An  example  of  a 
completely  open-ended  question  used  in  this  research  was  a 
question  asking  for  a  definition  of  the  term  risk.  Labaw 
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states  the  following  concerning  the  use  of  open-ended 

questions  which  concern  complex  issues: 

Open-ended  questions  provide  absolutely 
indispensable  insight  into  how  respondents 
interpret  complex  but  apparently  single-issue 
questions.  (25:134) 

op«  would  define  risk  is  a  prime  example  of  a 
complex  but  apparently  single-issue  question.  Although  risk 
is  a  seemingly  single  issue,  many  aspects  of  risk  exist 
which  can  complicate  the  task  of  defining  the  term. 

One  intent  of  the  open-ended  question  format  was  to 
allow  for  the  freedom  of  respondents  to  address  the  aspects 
of  risk  in  such  a  way  that  perceptions  about  risk  might  be 
categorized  according  to  risk  type  (cost,  schedule,  or 
performance).  Respondents  were  asked  to  provide  an 
assessment  of  what  they  perceived  their  own  risk  taking 
tendencies  to  be.  Questions  were  asked  to  ascertain  the 
relative  importance  of  risk  in  the  management  of  a  program. 
Along  with  these  open-ended  questions  were  a  series  of 
closed  questions  which  ask  about  specific  involvement  in 
various  program  management  areas.  Questions  were  also  asked 
to  determine  the  interview  subjects'  personal,  educational, 
and  work  backgrounds  to  see  if  they  correlated  in  any  way 
with  their  risk  taking  tendencies  (25:131). 

Emory  states  that  testing  of  the  interview  guide  is 
essential  to  ensure  that  the  interviewer  is  able  to  gather 
the  information  he  or  she  desires.  He  suggests  bringing 
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colleagues  into  the  testing  process  and  performing  a  "test- 
revise-retest  ”  cycle  of  tests  and  updates  (14:206-207). 

This  interview  guide  was  tested  through  a  series  of 
five  test  interviews  with  selected  Air  Force  Institute  of 
Technology  School  of  Systems  and  Logistics  students  and 
faculty.  These  students  and  faculty  members  all  had  former 
program  management  experience  and  were  all  familiar  with  the 
concept  of  risk  and  the  ways  it  influences  the  Air  Force 
acquisition  process.  Revisions  were  then  made  based  on 
recommendations  from  the  interviewees. 

IntervisH.  Guide 

The  interview  guide  (Appendix  A)  contained  two 
sections.  The  first  section  addressed  various  personal 
characteristics  of  the  interview  subject.  Data  in  this 
section  included  personal  data:  rank  or  grade,  age,  sex, 
marital  status,  and  number  of  children.  This  section  also 
asked  for  the  subject's  educational  history  and  work 
history.  The  data  from  this  first  section  was  used  to 
determine  whether  or  not  any  correlations  existed  between  a 
subject's  attitudes  toward  risk  and  any  of  the  five  program 
management  areas  outlined  earlier. 

The  second  section  of  the  interview  guide  contained  a 
series  of  13  questions  concerning  risk  and  its  relationship 
to  the  Air  Force  acquisition  environment.  Most  of  the 
questions  were  open-ended,  allowing  for  a  broad  range  of 


33 


responses.  Four  of  the  questions  were  closed  questions, 
requiring  a  specific  yes/no  or  numerical  response. 

The  following  section  describes  the  methods  of  analysis 
used  in  reducing  the  data  obtained  from  the  series  of 
interviews.  A  description  of  the  analysis  techniques  used 
for  each  of  the  interview  guide  questions  and  how  each 
question  applies  to  the  objectives  of  this  research  is 
discussed . 

Data-Analysis  and  Reduction 

The  data  obtained  through  the  25  interviews  were 
analyzed  to  answer  the  investigative  questions  outlined  in 
Chapter  I.  Questions  1,  9,  11,  and  12  of  the  interview 
guide  (Appendix  A)  were  asked  to  answer  the  first 
investigative  question  concerning  how  managers  view  the 
concept  of  risk.  How  did  they  define  risk?  How  did  they 
view  the  risk  assessments  that  were  being  performed?  How 
did  they  view  themselves  as  far  as  risk  was  concerned?  How 
did  they  view  their  own  knowledge  of  risk? 

Responses  to  question  1  were  categorized  by  whether  or 
not  the  interview  subjects  mentioned  the  consequence  element 
of  risk,  the  probability  element  of  risk,  both  elements,  or 
neither  element  in  their  definitions.  Category  totals  and 
percentages  were  calculated  and  graphed  to  get  a  feel  for 
how  well  the  definition  of  the  concept  of  risk  was 
understood . 
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Question  9  asked  the  respondents  to  rate  how  thoroughly 
they  felt  risk  assessments  and  analyses  were  performed  at 
ASD.  A  zero  to  one  hundred  scale  was  provided,  and  the 
respondents  were  asked  to  indicate  at  what  point  on  that 
scale  the  answer  to  the  question  should  fall.  The  mean  and 
variance  were  then  calculated  to  get  a  feel  for  how  ASD 
viewed  the  thoroughness  of  risk  assessments  and  analyses  in 
general . 

Question  11  asked  the  interview  subjects  to  describe 
themselves  as  far  as  risk  was  concerned  while  on  the  job. 
They  were  given  three  categories,  risk  taker,  risk  neutral, 
and  risk  avoiding.  Explanations  of  the  terms  risk  taker, 
risk  neutral,  and  risk  avoiding  were  given  to  interview 
subjects  who  indicated  that  they  were  unfamiliar  with  the 
terms.  The  explanations  used  were  the  explanations  of  these 
terms  described  in  Chapter  II.  The  responses  were 
categorized,  and  totals  were  calculated  and  graphed.  The 
graphs  were  created  to  display  possible  trends  in  responses. 
Also,  a  correlation  matrix  was  created  using  the  data  from 
question  11  and  the  personal  data  from  the  first  section  of 
the  interview  guide.  A  correlation  matrix  is  "the  tabular 
arrangement  of  all  the  correlation  coefficients  that  can  be 
calculated  from  all  the  possible  pairings"  of  the  data 
(21:223).  Using  the  STORM  software  package  for  quantitative 
decision  support,  values  of  r,  the  correlation  coefficient, 
were  calculated,  and  the  correlation  matrix  was  produced 
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(13:281).  The  correlation  matrix  compared  the  responses  of 
the  program  managers  as  risk  takers  with  the  following 
independent  variables: 

1.  Academic  Degree. 

2.  Types  of  Work  Experences . 

3.  Age  Greater  than  30. 

4.  Marital  Status. 

The  data  above  are  contained  in  Appendix  C. 

The  correlation  matrix  can  be  found  in  Appendix  D.  Any 
relationships  between  variables  are  discussed  in  Chapter  IV. 

Question  12  asked  the  interview  subjects  about  how 
comfortable  they  were  with  their  knowledge  of  risk  and  risk 
assessments.  Responses  were  categorized  by  yes,  no,  and 
unsure,  and  totals  were  calculated  and  graphed.  As  with  the 
question  11  graph,  the  question  12  graph  was  used  to  display 
any  trends  in  responses. 

Questions  6  and  7  were  asked  to  answer  the  second 
investigative  question  concerning  the  amount  and  types  of 
risk  program  managers  felt  were  present  in  each  of  the  five 
program  management  areas  outlined  previously.  Question  6 
simply  asked  the  respondents  to  indicate  whether  or  not  they 
had  been  involved  with  any  decision  making  involving  risk  in 
any  of  those  areas.  These  results  were  tabulated,  totaled, 
and  graphed. 

For  question  7,  interview  subjects  were  asked  to  rate 
the  cost,  schedule,  and  technical  performance  risks 


associated  with  each  of  the  five  areas  discussed  above  in 
question  6.  A  five  point  Likert  scale,  composed  of  five 
levels  of  importance,  ranging  from  "not  at  all"  to 
"extremely"  important,  was  used  with  this  question. 

Subjects  were  provided  with  a  matrix  of  all  15  individual 
categories  and  rated  each  using  the  scale.  Mean  values  for 
each  of  the  15  categories  were  calculated  and  tabulated. 

Data  from  question  7  were  also  used  to  answer  the  fifth 
investigative  question  concerning  possible  correlations 
between  risk  taking  tendencies  and  the  personal  data  from 
the  first  section  of  the  interview  guide.  Correlation 
matrices  were  created  using  these  data.  Three  correlation 
matrices  were  produced.  Each  correlation  matrix  compared 
the  responses  in  a  specific  risk  category  (cost,  schedule, 
or  technical  performance)  with  the  following  indicator 
variables ; 

1.  Academic  Degree. 

2.  Types  of  Work  Experiences. 

3.  Age  Greater  than  30, 

4.  Marital  Status. 

The  correlation  matrices  can  be  found  in  Appendix  E. 
Correlation  coefficients  S  0.4  were  analyzed,  and  these  are 
discussed  in  Chapter  IV. 

Questions  2  and  3  of  the  interview  guide  were  asked  to 
answer  the  third  investigative  question  which  asked  about 
the  methods  ASD  program  managers  used  to  evaluate  risk. 
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Question  2  asked  how  program  managers  usually  evaluated 
risk,  and  question  3  asked  if  they  used  any  other  methods 
than  those  discussed  in  question  2. 

Responses  to  question  2  were  categorized  in  two  ways. 
First,  they  were  categorized  as  to  whether  the  risk 
assessments  were  formal  versus  informal.  Second,  they  were 
categorized  as  to  whether  the  risk  assessments  were 
quantitative  versus  qualitative.  These  categories  were 
totaled  and  graphed. 

Statistical  tests  were  run  on  these  data  to  determine 
if  program  managers  tended  to  perform  informal  rather  than 
formal  or  qualitative  rather  than  quantitative  risk 
assessments.  A  binomial  test  was  used  because  the  data 
satisfied  the  requirements  of  independent  Bernoulli  trials. 
Those  assumptions  were  the  following: 

1.  The  outcomes  of  each  trial  can  be  classified 
as  a  "success"  or  a  "failure." 

2.  The  probability  of  a  success,  denoted  by  p, 
remains  constant  from  trial  to  trial. 

3.  The  n  trials  are  independent.  (19:15) 

Responses  to  the  question  were  dichotomous  and  could  be 
categorized  as  "successes,"  informal  and  qualitative,  or 
"failures,"  formal  and  quantitative.  There  was  no  reason  to 
believe  that  p  varied  from  trial  to  trial.  Finally,  each  of 
the  n  trials  was  independent. 

The  null  hypothesis  of  each  binomial  test  was  that  the 
probability  of  a  success  was  equal  to  the  probability  of  a 
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failure  (p  =  0.50).  In  other  words,  it  was  initially 
assumed  that  the  probability  of  performing  an  informal 
assessment  was  equal  to  the  probability  of  performing  a 
formal  assessment.  Similarly,  the  probability  of  performing 
a  qualitative  assessment  was  equal  to  the  probability  of 
performing  a  quantitative  assessment. 

The  alternative  hypothesis  of  each  test  was  that  p  was 
greater  than  0.5  and  that  ASD  program  managers  were  more 
likely  to  perform  informal  and  qualitative  risk  assessments 
than  formal  and  quantitative  risk  assessments  respectively. 

The  test  statistic  used  for  this  particular  test  was  B, 
the  number  of  successes  (informal  or  qualitative  risk 
assessments).  The  null  hypothesis  would  be  rejected  in 
favor  of  the  alternative  hypothesis  if  B  were  greater  than 
or  equal  to  b,  the  upper  tail  percentile  point  (number  of 
successes)  of  the  binomial  distribution  for  n  -  2b  and  p  = 
0.50.  The  value  of  b  was  obtained  from  an  upper  tail 
probability  table  for  binomial  disributions  (19:263). 

An  alternaLive  test  statistic  was  P,  the  probability 
that  B,  the  number  of  successes,  occurred  strictly  by  chance 
using  the  binomial  distribution  for  n  -  2b  and  p  =  0.50. 

This  value  was  also  obtained  from  an  upper  tail  probability 
table  for  binomial  distributions  (19:263).  If  this 
probability  was  less  than  0.05,  the  null  hypothesis  would  be 
rejected  in  favor  of  the  alternative  hypothesis  which  was 
that  p  was  greater  than  0.50. 
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Tests  were  run  using  both  of  the  above  test  statistics 
and  results  are  provided  in  Chapter  IV.  In  addition  to  the 
above  binomial  tests,  an  estimation  of  the  true  value  of  p 
(95%  confidence)  was  obtained  from  a  binomial  confidence 
limit  table  (32:454). 

The  fourth  investigative  question  was  asked  to 
determine  why  program  managers  performed  risk  assessments  in 
any  particular  manner.  Question  10  of  the  interview  guide 
addressed  this  investigative  question  directly.  A 
discussion  of  the  responses  can  be  found  in  Chapter  IV. 

Questions  4,  5,  and  13  of  the  interview  guide  addressed 
the  interview  subjects'  backgrounds  on  where  risk  assessment 
methods  were  learned,  what  quantitative  tools  they  were 
familiar  with,  and  what  guidance  on  risk  assessment  they 
were  familiar  with.  Discussions  of  their  responses, 
including  graphical  summaries  are  contained  in  Chapter  IV. 

Lijait.at-iQQS 

The  sample  was  taken  strictly  from  the  population  of 
program  managers  located  at  the  program  offices  of  ASD,  and 
should  not  be  generalized  to  represent  the  population  of 
program  managers  Air  Force  wide.  Also,  as  stated  earlier, 
the  sample  was  not  a  completely  random  sample  of  program 
managers  at  ASD.  The  judgement,  purposive  sample  may  be 
limited  in  its  generalizability  and  should  only  be  used  to 
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obtain  a  "feel"  for  ASD  program  managers'  views  of  risk  and 
risk  assessments. 


The  following  chapter  provides  an  analysis  of  the  data 
obtained  through  the  series  of  25  interviews  discussed 
earlier  in  this  chapter. 
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This  chapter  provides  an  analysis  of  the  data  collected 
for  this  research.  A  descriptive  summary  of  the  responses 
to  each  interview  question  is  provided.  This  summary 
includes  results  from  any  statistical  tests  performed  on  the 
responses.  Conclusions  and  recommendations  are  contained  in 
Chapter  V. 

Interview  Procedures 

Interviews  were  conducted  with  mid-level  Air  Force 
acquisition  program  managers  in  accordance  with  the  research 
design  outlined  in  Chapter  III.  Twenty-five  interviews  were 
conducted.  Interviewees  were  selected  using  a  purposive 
sample  of  program  management  personnel.  Twenty-one  military 
and  four  civilian  personnel  were  included  in  the  sample,  and 
the  sample  included  personnel  from  Air  Force  programs  at 
various  stages  of  development.  The  programs  ranged  from 
relatively  simple,  low  cost  programs  like  the  Laser  Eye 
Protection  program  being  developed  at  the  Life  Support 
System  Program  Office  (SPO),  to  technically  complex,  multi¬ 
million  dollar  programs  like  the  C-17  development  program. 
The  sample  was  considered  representative  of  the  personnel 
working  the  broad  spectrum  of  systems  being  developed  at  the 
Aeronautical  Systems  Division  (ASD). 

The  program  management  personnel  who  participated  in 
this  research  as  interview  subjects  appeared  to  have  put  a 
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great  deal  of  time  and  thought  into  their  responses  and  were 


all  highly  cooperative  and  accommodating  for  the  interviews. 
Each  interview  subject  was  asked  to  review  the  interview 
guide  prior  to  the  interview.  Some  subjects  had  prepared 
written  responses  to  some  of  the  questions  after  reviewing 
the  interview  guide.  The  interviews  lasted  an  average  of 
approximately  25  minutes.  A  detailed  summary  of  responses 
to  selected  questions  can  be  found  in  Appendix  B. 

Interview  Questions 

Question  1.  HOW  WOULD  YOU  DESCRIBE  THE  CONCEPT  OF 

RISK? 


This  question  was  asked  to  get  a  feel  for  how  program 
managers  viewed  the  concept  of  risk.  How  was  the  program 
manager's  view  of  risk  different  from  or  like  the  views 
brought  forth  in  the  literature  on  risk  and  risk  assessment? 

Responses  to  this  question,  as  expected,  were  highly 
variable.  As  stated  in  Chapter  II,  the  DSMC  definition  of 
risk  involves  both  the  probability  and  the  significance  of 
the  consequence  of  an  undesirable  event.  Twelve  of  25 
responses  mentioned  the  consequence  of  a  decision  in  their 
description  of  risk.  However,  of  these  12,  only  seven 
subjects  suggested  that  the  significance  of  the  consequence 
was  a  determinant  of  the  level  of  risk.  Ten  of  25  responses 
suggested  that  risk  involved  the  chance,  likelihood,  or 
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probability  of  occurrence  of  an  undesirable  outcome.  Five 

of  the  responses  included  both  the  probability  of  occurrence 

and  the  consequence  of  an  event.  Eight  responses  said 

nothing  about  probabilities  of  occurrence  or  consequences. 

Some  of  these  responses  stressed  the  uncertainty 

associated  with  a  situation  as  the  element  of  risk.  This 

view  is  represented  in  the  following  response: 

Risk  is  the  uncertainty  induced  by  schedule  or 
budget  problems  caused  by  unrealistic 
expectations.  Risk  arises  when  proposed  schedules 
are  unrealistic.  Our  risks  are  generally  low 
because  we  don't  really  push  the  state-of-the-art. 

Certain  subjects  felt  the  term  risk  was  vague  and 

ambiguous.  They  had  a  difficult  time  coming  up  with  a 

definition.  These  definitions  seemed  to  describe  where  risk 

was  addressed,  but  not  what  it  actually  was.  This  is  shown 

in  the  following  definition  of  risk: 

The  risk  of  a  program  depends  on  the  technical 
aspects  of  the  program,  the  dollars  spent,  and  the 
schedule  impacts  of  certain  decisions.  Risk  is 
subjective  The  user  of  the  weapon  system  drives 
the  requirements  and  the  subsequent  risks. 

Both  of  the  above  responses  discuss  possible  causes  of 

risk.  Unrealistic  schedules,  technically  complex 

requirements,  and  limited  funding  cause  the  level  of  risk  to 

increase.  Though  these  responses  indicate  possible  causes 

of  risk,  they  do  not  actually  define  what  risk  is. 

Others  were  more  familiar  with  risk,  but  still  had 

problems  providing  a  clear,  succinct  definition.  This  is 

shown  in  the  definition  below: 
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Risk  is  the  degree  at  which  you'll  expect  to 
accomplish  sometliing  within  constraints  (money, 
time,  resources,  technology).  Risk  is  derived 
from  uncertainty.  At  work,  the  degree  of 
consequence  is  the  driver. 

The  responses  to  question  1  support  the  literature  in 
that  there  is  no  standardized  DoD  definition  of  risk  present 
in  the  responses  of  these  program  managers.  The  data 
suggests  that  risk  is  viewed  in  a  number  of  ways  with 
varying  degrees  of  specificity  as  far  as  probabilities  and 
significance  of  consequences  are  concerned.  A  summary  of 
which  concepts  were  mentioned  by  whom  in  defining  risk  is 
shown  in  Table  1.  The  data  also  indicate  that  the  focus  of 
many  program  managers  is  on  the  causes  of  risk.  This  may  be 
advantageous  in  that  program  managers  might  be  better  able 
to  reduce  risk,  if  they  are  aware  of  the  causes. 

Question  2.  DESCRIBE  HOW  YOU  USUALLY  EVALUATE  RISK.  WHAT 
PROCESS  OR  STEPS  DO  YOU  GO  THROUGH?  CAN  YOU  GIVE  AN 
EXAMPLE? 

This  question  was  asked  to  determine  how  program 
managers  evaluate  risk.  Do  they  use  some  sort  of  formal 
process?  Do  they  lean  toward  qualitative  methods  versus 
quantitative  methods? 

Table  2  is  a  summary  of  responses  categorizing  the  risk 
assessment  processes  of  ASD  program  managers  to  be  formal  or 
informal  and  quantitative  or  qualitative.  For  the  purposes 
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Table  1 


Risk  Definition  Summary  (Question  1) 


Interview 

Number 

(A) 

Consequence 

<B> 

Consequence 

Significance 

1  (C) 

1  Probability  1 

1  I 

1  1 

1  1 

Both 

A&C 

1 

X 

X 

X 

2 

3 

4 

X 

X 

X 

X 

5 

X 

6 

X 

7 

X 

X 

X 

X 

8 

X 

9 

X 

10 

X 

11 

X 

X 

12 

13 

X 

14 

X 

15 

16 

17 

X 

X 

X 

X 

18 

X 

X 

19 

20 

21 

X 

22 

X 

23 

X 

X 

X 

X 

24 

25 

X 

X 

Totals 

12  (48%) 

7  (28%) 

o 

o 

5  (20%) 

of  this  research,  a  formal  assessment  was  defined  to  be  any 
standard  process  a  program  manager  went  through  to  assess 
risk,  be  it  quantitative  or  qualitative.  This  process  was 
the  same  process  he  or  she  went  through  each  time  he  or  she 
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Table  2 


Risk  Assessment  Type  Summary  (Question  2) 


t 

1 

Interview  |  Formal 

Number  1 

1 

1  1 

1  1 

Informal  1  Qualitative  1  Quantitative 

t  1 

t  1 

1  t 

1 

X  X 

2  X 

X 

3 

X  X 

4 

X  X 

5 

X  X 

6 

X  X 

7 

X  X 

8  X 

X 

g 

X  X 

10 

X  X 

11 

X  X 

12 

X  X 

13 

X  X 

14 

X  X 

15 

X  X 

16 

X  X 

17 

X  X 

18  X 

X 

19 

X  X 

20 

X  X 

21 

X  X 

22 

X  X 

23 

X  X 

24 

X  X 

25 

X  X 

Totals  !  3  (12%) 

1 

1  t 

22  (88%)  :  22  (88%)  '  3  (12%) 

1  1 

«  1 

performed  a  risk  assessment.  Conversely,  informal  risk 
assessments  had  no  specific  structure  or  method. 

A  quantitative  risk  assessment  was  defined  to  be  any 
process  in  which  some  kind  of  numerical  process  was  used  to 
generate  an  answer  which  described  the  risk  of  a  situation. 


The  quantitative  assessment  was  generally  a  more  objective 
assessment.  A  qualitative  assessment  was  classified  as  any 
non-numerically  based  assessment  and  was  generally  more 
sub j  ect ive . 

An  overwhelming  majority,  22  of  25  (88%),  of  interview 

subjects  performed  informal,  qualitative  risk  assessments. 

For  each  interview  subject,  a  risk  assessment  consisted  of 

an  information  gathering  exercise  from  which  the  risk 

assessment  was  drawn.  Seventeen  of  25  responses  (68%) 

mentioned  that  they  would  gather  information  by  seeking 

advice  from  government  experts  when  making  a  risk 

assessment.  Ten  of  25  subjects  (40%)  said  that  they  relied 

on  their  own  past  experience  when  performing  a  risk 

assessment.  Other  sources  of  information  were  historical 

(similar)  programs,  lessons  learned  files,  and  contractor 

supplied  information.  Several  respondents  mentioned  the 

politics  involved  with  performing  risk  assessments.  For 

some,  these  political  pressures  proved  to  be  quite  high,  as 

with  the  following  response: 

Political  influences  are  very  strong  and  can  cause 
a  legitimate  risk  assessment  to  be  thrown  out  the 
window  so  that  a  more  favorable  assessment  can  be 
briefed  up  the  chain. 

Table  2  also  suggests  that  the  interviewees  may  have 
perceived  the  terms  "informal"  and  "qualitative"  in  a 
similar  way.  Likewise,  the  terms  "formal"  and 
"quantitative"  may  have  been  perceived  as  being  similar  Of 
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the  25  responses,  23  were  paired  in  this  way  (all  but 
Interview  Numbers  2  and  14). 

Binomial  tests  of  these  data  verify  the  fact  that  ASD 
program  managers  are  more  likely  to  perform  informal, 
qualitative  risk  assessments  than  formal,  quantitative  risk 
assessments.  The  binomial  tests  performed  on  the  data  show 
two  things  to  support  the  above  conclusion.  First,  the 
number  of  "successes"  or  informal,  qualitative  risk 
assessments  that  would  suggest  an  equal  likelihood  (p  = 
0.50)  of  f ormal/ inf ormal  or  quantitative/qualitative 
assessments  should  be  close  to  13.  Since  B  =  22,  the  data 
suggests  that  p  is  actually  much  greater  than  0.50. 
Secondly,  from  a  table  of  upper  tail  probabilities  for 
binomial  distributions,  the  probability,  P,  that  22  of  25 
persons  would  do  informal,  qualitative  risk  assessments 
given  an  equal  chance  of  doing  formal  or  informal  and 
quantitative  or  qualitative  assessments  is  «  0.0001 
(19:263).  Again,  this  provides  strong  evidence  that  p  is 
actually  much  greater  than  0.50.  A  95%  confidence  interval 
on  the  true  value  of  p  was  obtained  from  a  binomial 
confidence  limit  table  and  verifies  that  p  is  actually 
greater  than  0.50  (32:454).  The  lower  and  upper  bounds 
respectively  of  the  true  value  of  p  are  0.6876  and  0.9745 
(0.6876  <  p  <  0.9745).  These  binomial  tests  are  described 
in  detail  in  Chapter  III,  and  a  summary  of  the  results  is 
contained  in  Table  3. 
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Table  3 


Binomial  Test  Results 

Informal  vs  Formal  and  Qualitative  vs  Quantitative 


Null  Hypothesis  Alternative  Hypothesis 
Ho:  p  =  0.50  Ha:  p  >  0.50 

Test  Statistic:  B  (Number  of  Informal  or  Qualitative 

Assessments ) 

Rejection  Region:  Reject  Ho  if  B  S  b 

Where:  b  (The  upper  tail  percentile  point  of  the  binomial 

distribution  for  77=25  and  p=.5) 

Results:  (For  Both  Tests) 

b  =  13 
B  =  22 
B  >  b 

Therefore:  Reject  Ho,  and , p  >  0.50. 

Alternative  Test: 

Test  Statistic:  P  (Probability  that  B,  the  number  of 

successes,  occurred  strictly  by  chance  for  77=25  and  p= .  5 ) 

Rejection  Region:  Reject  Ho  if  P  i  0.05 

Results:  (For  Both  Tests) 

P  =  0.0001 
P  <  0.05 

Therefore:  Reject  Ho,  and  p  >  0.50. 

From  the  binomial  confidence  limit  table  (95%  confidence): 
p  Lower  =  0.6876  p  Upper  =  0.9745 


(0.6876  <  p  <  0.9745) 


Question  3.  DO  YOU  USE  ANY  OTHER  METHODS  TO  EVALUATE  RISK? 
IF  SO,  WHICH  ONES? 

Since  question  2  asked  how  acquisition  program  managers 
usually  evaluated  risk,  question  3  was  included  to  determine 
what  other  methods,  if  any,  the  subjects  used  when 
performing  a  risk  assessment.  In  response  to  this  question, 
the  interview  subjects  seemed  to  be  inclined  to  discuss 
whether  or  not  they  had  used  any  of  the  quantitative  tools 
mentioned  in  question  5  of  the  interview  guide.  Fifteen  of 
25  respondents  said  that  they  used  no  other  methods  to  • 
evaluate  risk  than  those  discussed  in  response  to  question 
2.  Five  of  the  subjects  mentioned  using  a  network  analysis 
on  their  program,  but  only  one  of  those  five  specifically 
addressed  schedule  risks  with  the  analysis.  Two  subjects 
mentioned  using  life  cycle  cost  models,  and  one  person 
mentioned  using  Milestone  or  Gantt  type  charts  to  address 
schedule  risk.  These  responses  indicate  that  very  few 
explicit  methods  or  techniques  are  used  by  ASD  program 
managers  to  evaluate  risk. 

Question  4.  WHERE  DID  YOU  LEARN  THE  METHODS  DESCRIBED 
ABOVE? 

This  question  was  asked  to  determine  the  source  of  each 
program  manager's  knowledge  of  risk  and  risk  assessments. 
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Because  some  respondents  determined  that  they  had  no  real 


method  of  performing  a  risk  assessment,  they  felt  that  this 
question  was  not  applicable.  Seventeen  of  21  who  responded 
indicated  that  what  they  learned  about  risk  assessments  was 
learned  on  the  job  or  through  personal  experience.  Three 
subjects  indicated  that  they  learned  their  networking 
techniques  in  school.  One  respondent  indicated  that  the 
methods  were  learned  from  contracting  courses  and  the 
Federal  Acquisition  Regulation.  These  results 
indicate  that  risk  assessment  techniques  are  not  being 
acquired  through  formal  program  management  courses. 

Question  5.  ARE  YOU  FAMILIAR  WITH  ANY  OF  THE  FOLLOWING 
QUANTITATIVE  RISK  ASSESSMENT  TOOLS? 


1 . 

EXPECTED 

VALUE/UTILITY  ANALYSIS 

2. 

NETWORK  , 

ANALYSIS 

3. 

DECISION 

TREE  ANALYSIS 

4  . 

LIFE  CYCLE  COST  MODELS 

5. 

COMPUTER 

SIMULATION 

Question  5  was  asked  to  determine  which  quantitative 
risk  assessment  tools  ASD  program  managers  had  some 
familiarity  with.  The  reasoning  was  that  a  quantitative 
risk  assessment  was  not  possible  without  the  proper  tools 
and  training.  Results  showed  that  all  interview  subjects 
were  familiar  with  at  least  one  type  of  network  analysis 
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technique.  Twenty  of  25  (80%)  were  familiar  with  Life  Cycle 
Cost  (LCC)  models,  and  a  majority  were  familiar  with 
decision  tree  analysis  (68%).  Less  than  half  of  the 
respondents  indicated  any  familiarity  with  expected  value 
analysis  (40%)  or  computer  simulation  techniques  (36%) 
(Figure  2).  These  results  indicate  that  the  reason  for  a 
lack  of  quantitative  risk  assessments  is  not  necessarily  a 
lack  of  knowledge  of  risk  assessment  tools. 

Question  6.  HAVE  YOU  BEEN  INVOLVED  WITH  A  RISK  ASSESSMENT 
IN  MAKING  DECISIONS  AFFECTING  ANY  OF  THE  FOLLOWING  AREAS? 

1.  CONTRACTOR  SOURCE  SELECTION 

2.  PREPARING  AN  ACQUISITION  STRATEGY 

3.  PREPARING  AN  ACQUISITION  PLAN  RISK  ASSESSMENT  FOR 
THE  PROCURING  CONTRACTING  OFFICER 

4.  DETERMINING  THE  TYPE  OF  CONTRACT  TO  BE  USED 

5.  DETERMINING  TEST  &  EVALUATION  OBJECTIVES 

6.  OTHER  AREAS 

Question  6  was  the  first  of  two  closed  type  questions 
which  specifically  addressed  the  objective  of  this  research. 
That  objective,  as  stated  in  Chapter  I,  was  to  determine  how 
and  to  what  degree  Air  Force  program  managers  at  ASD 
evaluate  the  risk  associated  with  the  program  decision  areas 
listed  above.  This  question  simply  asked  whether  or  not  the 
interview  subject  had  been  involved  with  any  decis  . n  making 
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gure  2.  Quantitative  Assessment  Tools 


involving  risk  in  any  of  the  above  areas.  Results  are  shown 
in  Table  4  and  Figure  3. 

Each  subject  interviewed  had  been  involved  in  doing  a 
risk  assessment  for  making  decisions  in  at  least  one  of  the 
five  areas  listed.  For  each  of  the  five  program  management 
areas,  a  majority  of  the  interview  subjects  had  been 
involved  with  making  a  risk  assessment  and  decision. 
Contractor  Source  Selection  and  Acquisition  Strategy 
preparation  were  the  highest  on  the  list,  80%  and  76% 
involved  respectively.  These  were  followed  by  T&E 
Objectives  -  64%,  Contract  Type  -  60%,  and  Acquisition  Plan 
preparation  -  52%. 

These  results  indicate  that  risk  decisions  are  being 
made  by  the  majority  of  program  managers  in  all  of  the 
previously  described  program  management  areas.  The 
differences  in  the  above  percentages  may  be  caused  by  the 
fact  that  different  program  managers  have  had  different 
program  management  responsibilities.  Not  all  program 
managers  are  involved  with  every  phase  or  aspect  of  a 
program.  For  example,  a  program  manager  may  have  started 
working  with  a  program  after  the  contract  had  been  awarded. 
Consequently,  he  or  she  was  not  exposed  to  the  decision 
process  associated  with  the  type  of  contract  awarded.  This 
does  not,  however,  mean  that  this  program  manager  would  not 
be  concerned  with  the  risk  aspects  ov  such  a  decision  if  he 
or  she  were  exposed  to  such  a  decision  later  in  his  or  her 
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Table  4 


Risk  Assessment  Involvement  (Question  6) 


Interview 

Source 

Acq 

Acq 

Contract 

Test 

Subj  ect 

Selection 

Strat 

Plan 

Type 

Obj  ect ives 

1 

X 

X 

X 

X 

X 

2 

X 

X 

3 

X 

X 

X 

X 

4 

X 

X 

X 

X 

5 

X 

X 

6 

X 

X 

7 

X 

X 

X 

8 

X 

X 

9  • 

X 

X 

X 

X 

10 

X 

11 

X 

X 

12 

X 

X 

X 

X 

13 

X 

X 

X 

X 

14 

X 

X 

X 

X 

X 

15 

X 

X 

X 

X 

X 

16 

X 

X 

X 

X 

17 

X 

18 

X 

X 

X 

X 

19 

X 

X 

X 

X 

20 

X 

X 

X 

X 

21 

X 

X 

22 

X 

X 

X 

X 

23 

X 

X 

X 

X 

24 

X 

X 

X 

X 

25 

X 

X 

X 

Totals 

20 

19 

13 

15 

16 

(80%) 

(76%) 

(52%) 

(60%) 

(64%) 

career.  Question  7  followed  up  question  6  by  asking  how 
important  the  risks  involved  with  these  decisions  were  to 
the  program  managers.  The  reason  the  question  of  risk 
importance  was  asked  was  that  the  more  important  a  risk  was, 
the  more  emphasis  a  program  manager  would  likely  place  on  a 
particular  risk  assessment.  This  would  help  in  determining 
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Assessment  I nvo 1 vement 
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Figure  3.  Risk  Assessment  Involvement 


the  degree  of  risk  evaluation  associated  with  the  program 
decision  areas. 


Question  7.  PLEASE  INDICATE  IN  THE  MATRIX  BELOW  WHAT  TYPES 
OF  RISK  (COST,  SCHEDULE,  PERFORMANCE)  WERE  ASSOCIATED  WITH 
EACH  OF  THE  ABOVE  PROGRAM  DECISION  AREAS?  IN  THE  SAME 
BOXES,  PLEASE  INDICATE  THE  RELATIVE  IMPORTANCE  OF  EACH  OF 
THESE  RISK  TYPES  USING  THE  SCALE  BELOW  (1  THROUGH'  5). 


Not  at  all  Llttla  SooiauHat 

Important  Importance  Important 


Very  Extremely 

Important  Important 


Cost 

Schedule 

Source  Selection 

Acquisition  Strategy 

Acquisition  Plan 

Contract  Type 

T&E  Objectives 

Other 

Technical 

Performance 


58 


Question  7  was  asked  to  determine  the  types  and  the 
relative  importance  of  the  risks  the  program  managers  felt 


were  present  in  each  of  the  above  areas.  A  five  point 
Likert  scale  was  used  to  rate  the  relative  risk  importance. 
Responses  to  this  question  and  descriptive  statistics  are 
assembled  in  Appendix  C.  Mean  values  for  each  risk  category 
are  shown  below  in  Table  5.  (Note:  The  "Other”  category  is 
not  included  in  Table  5  because  of  the  small  number  and 
variety  of  responses.  However,  an  overall  mean  per  risk 
category  (columns)  and  per  program  management  area  (rows)  is 
provided . ) 


Table  5 

Risk  Importance  (Mean  Values)  (Question  7) 


Risk  Category 

Program 

Management  Area 

Cost 

Schedule 

Tech 

Perf 

Overall 
Per  Area 

Source  Selection 

4.00 

3.68 

4.45 

4.04 

Acquisition  Strategy 

3.64 

4 . 04 

3.84 

3 . 84 

Acquisition  Plan 

4 . 00 

4 . 08 

3.71 

3 . 93 

Contract  Type 

4.42 

3.46 

3 . 59 

3 . 82 

T&E  Objectives 

3.04 

3 . 70 

4 . 57 

3.77 

Overall  Per  Category 

3 . 82 

3.79 

4  03 

59 


Table  5  shows  no  consistent  pattern  of  a  high  or  low 
rating  of  risk  in  any  particular  risk  category  (cost, 
schedule,  or  performance)  or  for  any  particular  program 
management  decision  area.  Several  of  the  combinations  of 
risk  categories  and  program  management  decision  areas, 
however,  had  logical  highs  and  lows.  For  example,  because 
cost  is  the  primary  focus  of  most  contract  types,  cost  risk 
was  rated  highest  in  the  Contract  Type  area.  Schedule  risk 
was  rated  highest  in  the  two  planning  areas,  development  of 
the  Acquisition  Strategy  and  development  of  the  Acquisition 
Plan.  Technical  risk  was  rated  highest  in  the  two  areas  in 
which  system  performance  is  typically  stressed,  the  Source 
Selection  and  Test  &  Evaluation  areas. 

The  highest  overall  rating  of  risk  importance  was  with 
the  technical  performance  risk  associated  with  Test  & 
Evaluation.  It  follows  that  the  lowest  overall  rating  of 
risk  importance  was  with  the  cost  risk  associated  with  Test 
&  Evaluation  These  two  facts  suggest  that  program  managers 
may  tend  to  sacrifice  cost  risks  in  order  to  address 
technical  risks  during  the  test  and  evaluation  phase  of  a 
program . 

It  should  be  noted  that  subjects  were  asked  to  rate  the 
relative  risk  importance  (cost,  schedule,  and  performance) 
according  to  each  program  management  decision  area  (source 
selection,  acquisition  strategy,  etc.)  without  regard  to 
other  areas,  so  any  comparison  among  risk  areas  should  be 


viewed  with  caution.  In  other  words,  the  technical 
performance  risk  of  Test  &  Evaluation  Objectives  is  not 
necessarily  the  most  important  of  all  categories  because  the 
technical  risk  of  T&E  was  not  compared  with  the  technical 
risk  of,  for  example.  Source  Selection. 

Nine  of  the  interview  subjects  mentioned  other  areas  in 
which  risk  assessments  were  performed.  Two  subjects 
mentioned  the  fact  that  the  budgeting  process  contained  some 
inherent  risk.  Risk  was  associated  with  the  question,  "How 
can  you  get  funding?"  Alternative  methods  for  obtaining 
necessary  funding  were  analyzed,  and  this  analysis  included 
an  assessment  of  the  risk  associated  with  each  alternative. 

Five  subjects  mentioned  the  risk  associated  with  system 
design.  Of  these  five,  two  mentioned  the  risk  associated 
with  system  design  trade-offs  or  trade  studies.  Two 
subjects  pointed  out  the  risk  associated  with  contract 
modifications.  One  subject  mentioned  the  program  risk 
assessments  that  were  done  on  his  program  during  program 
design  reviews  and  technical  interchange  meetings.  In  each 
of  these  cases,  risk  was  a  factor  in  determininfi  system 
design . 

Other  program  management  areas  mentioned  in  which  risk 
assessments  were  performed  were  software  development,  cost 
estimates,  and  the  decisions  associated  with  providing  the 
defense  contractor  Government  Furnished  Equipment  (GFE). 
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A  correlation  analysis  was  done  on  the  data  from 
Question  7  and  the  data  collected  from  the  Personal  Data 
section  of  the  interview  guide.  Correlation  matrices  of  the 
Cost  Risk  Importance,  Schedule  Risk  Importance,  and 
Technical  Risk  Importance  are  contained  in  Appendix  E. 
Indicator  variables  included  in  each  correlation  matrix  were 
the  following: 

1.  Academic  Degree. 

2.  Types  of  Work  Experiences. 

3.  Age  Greater  than  30. 

5.  Marital  Status. 

The  correlation  analysis  was  done  to  determine  possible 
relationships  between  risk  taking  tendencies  and  the  above 
independent  variables.  All  correlations  with  an  absolute 
value  5  0.400  are  discussed  below. 

Three  items  in  the  Cost  Risk  correlation  matrix  had  a 
correlation  coefficient  >  0.400.  The  cost  risk  associated 
with  Contract  Type  indicated  a  direct  relationship  with  age 
(r  =  0.419).  The  subjects  over  30  years  of  age  were 
inclined  to  indicate  a  high  cost  risk  importance  rating  to 
contract  type. 

Two  items  under  Test  and  Evaluation  had  correlation 
coefficients  5  0.400.  An  inverse  relationship  was  found  to 
exist  between  degree  type  and  the  cost  risk  associated  with 
T&E  (r  =  -0.470).  Non-technical  degree  holders  tended  to 
indicate  higher  risk  ratings  associated  with  cost,  and 
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technical  degree  holders  tended  to  indicate  lower  ratings. 
Finally,  a  direct  relationship  was  found  to  exist  between 
having  some  operational  experience  and  the  cost  risk 
associated  with  T&E  (r  =  0.425).  Program  managers  with  some 
operational  experience,  as  defined  in  Chapter  I,  tended  to 
stress  the  cost  risks  associated  with  testing  and  evaluating 
weapon  systems. 

No  items  were  found  to  have  a  correlation  coefficient  > 
0.400  in  the  Schedule  Risk  correlation  matrix,  and  one  item 
had  a  correlation  coefficient  ^  0.400  in  the  Technical 
Performance  Risk  correlation  matrix.  That  item  indicated  an 
inverse  relationship  between  degree  type  and  the  technical 
risk  associated  with  Source  Selection  (r  =  -0.442).  Non¬ 
technical  degree  holders  tended  to  indicate  higher  technical 
perfomance  risk  ratings  for  the  source  selection  process. 

Question  8.  IF  RISK  ASSESSMENTS  WERE  DONE  IN  ANY  OF  THE 
ABOVE  AREAS,  WHAT  METHODS  WERE  USED? 

Responses  to  Question  8  turned  out  to  be  no  different 
than  the  responses  provided  for  Question  2.  The  reason  the 
question  was  asked  was  to  determine  whether  the  specific 
discussions  of  risk  assessment  tools  and  risk  assessment 
areas  called  to  mind  any  risk  assessment  methods  that  may 
have  been  used,  but  were  not  thought  of  previously.  None  of 
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the  interview  subjects  mentioned  any  other  methods  of  risk 


assessment . 

Question  9.  GENERALLY,  ON  A  SCALE  FROM  0  TO  100,  HOW 
THOROUGHLY  ARE  RISK  ASSESSMENTS  OR  ANALYSES  PERFORMED? 

Question  9  was  asked  to  get  a  feel  for  how  ASD  program 
managers  viewed  the  risk  assessments  and  analyses  that  were 
being  performed  at  ASD.  Specifically,  were  the  manners  in 
which  the  assessments  and  analyses  were  done  rigorous  and 
thorough;  were  they  superficial,  or  were  they  somewhere  in 
between? 

Responses  are  contained  in  Appendix  F.  The  mean  value 
of  the  25  responses  was  50.80.  Responses  ranged  from  a 
minimum  of  10  to  a  maximum  of  85.  In  general,  program 
managers  were  neutral  as  to  their  feelings  on  the 
thoroughness  of  risk  assessments  and  analyses  at  ASD.  Many 
subjects  provided  comments  along  with  their  numerical 
responses.  Some  subjects  indicated  that  a  less  than 
rigorous  risk  assessment  was  adequate  in  most  cases,  and 
performing  an  assessment  which  approached  100  on  the  scale 
would  provide  a  benefit  not  worth  the  cost.  Many  interview 
subjects  indicated  that  the  necessary  thoroughness  of  a  risk 
assessment  varied  with  the  perceived  risk  prior  to  the 
assessment.  If  the  potential  risk  was  perceived  to  be  high, 
then  a  thorough  risk  assessment  would  be  performed. 
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Question  10.  WHAT  IS  THE  PRIMARY  REASON  YOU  PERFORM  RISK 


ASSESSMENTS  THE  WAY  YOU  DO? 

Question  10  was  asked  to  determine  the  circumstances 
that  lead  the  program  managers  to  evaluate  risk  in  the  ways 
discussed  previously.  Were  the  program  managers  directed  by 
superiors  to  perform  the  analyses  in  a  specified  manner? 

Was  time  a  factor  in  the  choice  of  a  risk  evaluation  method? 

Responses  to  this  question  varied.  Time  constraints 
were  the  primary  reason  for  nine  of  the  25  interview 
subjects.  Six  of  the  subjects  indicated  that  they  did  risk 
assessments  the  way  they  did  them  because  it  was  the  only 
way  they  knew  how,  but  it  seemed  to  be  adequate.  One 
subject  indicated  that  he  did  them  because  of  tasking  and 
that  the  level  of  risk  was  usually  pre-determined 
politically. 

Seven  of  the  responses  addressed  the  question  more  from 
the  angle  of  why  risk  assessments  were  done  in  general, 
rather  than  why  they  performed  risk  assessments  in  the  way 
that  they  did.  Two  typical  responses  from  these  individuals 
are  given  below. 

1.  I  do  them  to  answer  questions  I  have  about 

schedule,  cost,  and  performance. 

2.  To  reach  the  best  possible,  cost  effective, 

optimum  solution  to  a  problem. 
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Question  11.  HOW  WOULD  YOU  DESCRIBE  YOURSELF  AS  FAR  AS  RISK 


IS  CONCERNED?  RISK  TAKER?  RISK  NEUTRAL?  RISK  AVOIDING? 
WHY? 

Question  11  was  asked  to  get  a  feel  for  how  ASD  program 
managers  viewed  themselves  as  risk  takers  at  work.  A 
summary  of  responses  according  to  category  is  shown  in 
Figure  4. 

There  were  some  interesting  responses  to  this  question. 
Many  subjects  indicated  that  their  risk  taking  tendencies  on 
the  job  differed  from  their  risk  taking  tendencies  in  their 
personal  life.  Some  interview  subjects  were  willing  to  take 
more  risks  on  the  job.  Conversely,  some  were  willing  to 
take  more  personal  risk.  The  following  responses  illustrate 
these  views. 

1.  Personal  risk  -  risk  taker.  These  are  known 
quantities.  Professionally  -  risk  averse.  I'm 
dealing  with  government  money.  I  can't  gamble 
with  what  doesn't  belong  to  me. 

2.  Personally,  risk  taker.  You  need  to  rake 
risks  to  get  anywhere.  On  tiie  job,  risk  neutral 
because  of  the  environment. 

•3.  I  value  my  own  security.  I'll  take  more  risks 
at  work  than  with  my  personal  life. 

4.  Personally,  not  a  risk  taker.  Professionally, 
you  need  to  take  risks.  You  have  to  take  risks  to 
get  things  done.  It's  very  much  a  function  of  the 
environment . 

Many  of  the  interview  subjects  pointed  out  that  risk 
taking  behavior  was  dependent  on  the  situation  or  the 
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NumDer  of  Responses 


environment.  For  example,  "I  take  risks  when  the  need  is 
urgent . " 

There  seemed  to  be  a  general  feeling  that  in  order  to 
get  anything  accomplished  at  work,  risks  had  to  be  taken. 
This  was  described  by  one  interview  subject  as  the  "nothing 
ventured,  nothing  gained"  philosophy. 

A  correlation  analysis  was  done  comparing  the  program 
managers'  responses  to  question  11  and  the  indicator 
variables  described  above  in  the  analysis  of  question  7 
(Appendix  D).  The  largest  value  of  r,  the  correlation 
coefficient,  produced  by  the  correlation  analysis  was  0.250. 
This  was  the  relationship  between  possessing  a  technical 
degree  and  describing  oneself  as  a  risk  taker.  Because  it 
is  less  than  the  0.40  used  in  this  study,  the  reader  should 
decide  whether  or  not  this  relationship  is  significant. 

Question  12.  ARE  YOU  COMFORTABLE  WITH  YOUR  KNOWLEDGE  OF 
RISK  AND  RISK  ASSESSMENTS?  IF  NOT,  IN  WHICH  AREAS  WOULD  YOU 
LIKE  TO  INCREASE  YOUR  KNOWLEDGE? 

Question  12  was  asked  to  get  a  feeling  for  how  content 
ASD  program  managers  were  with  their  knowledge  of  the 
concept  of  risk  and  their  ability  to  perform  adequate  risk 
assessments.  Figure  5  shows  how  the  interview  subjects 
responded  to  this  question. 
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Knowledge  of  Risk 


These  results  show  no  general  feeling  one  way  or  the 
other  as  to  whether  ASD  program  managers  are  comfortable 
with  their  knowledge  of  risk  and  risk  assessments.  However, 
the  fact  that  12  of  25  program  managers  (48%)  said  that  they 
were  uncomfortable  with  their  knowleage  of  risk  assessments 
indicates  that  more  education  and/or  experience  may  be 
necessary.  In  fact,  seventeen  of  25  responses  (68%) 
indicated  a  need  for  additional  information  about  some 
aspect  of  risk.  Of  these  17,  seven  mentioned  that  they 
would  like  to  learn  more  about  some  of  the  quantitative  risk 
assessment  tools  pointed  out  in  Question  5.  Others  would 
like  to  learn  more  about  some  of  the  formal  risk  assessment 
processes  (net  necessarily  quantitative).  Still  others 
stated  that  additional  experience  would  be  useful. 

Question  13.  ARE  YOU  FAMILIAR  WITH  ANY  DOD  GUIDANCE  ON  RISK 
ASSESSMENT,  ANALYSIS,  AND  MANAGEMENT?  IF  SO,  WHICH  GUIDANCE? 
WHERE  WOULD  YOU  LIKE  TO  SEE  MORE  GUIDANCE? 

Question  13  was  the  last  question  asked  of  the 
interview  subjects,  and  it  was  asked  to  determine  what 
Department  of  Defense  sponsored  information  was  in 
circulation  within  ASD  on  risk  or  any  subject  relating  to 
risk.  Only  eight  subjects  mentioned  any  knowledge  of  DoD 
guidance  relating  to  risk.  Tvro  of  those  eight  mentioned 
knowledge  of  the  Willoughby  Templates  for  identifying  risk 


7C 


areas.  These  templates  are  similar  to  those  found  in  the 
manual  accompan: ing  DoD  Directive  4245.7  (Discussed  in 
Chapter  II).  One  subject  mentioned  knowledge  of  Military 
Standard  882B,  System  Safety  Proi^ram  Requirements.  This  is 
a  standard  which  provides  guidance  on  flight  safety  issues, 
and  it  was  used  by  this  particular  individual  during  his 
time  at  the  Air  Force  Flight  Test  Center,  Edwards  Air  Force 
Base,  California.  One  subject  mentioned  Air  Force  Systems 
Command  Regulation  70-15,  the  Air  Force  guide  to  contractor 
source  selection.  Two  of  the  eight  lesoonses  indicating 
knowledge  of  DoD  guidance  on  risk  were  not  specific  as  to 
what  guidince  they  were  familiar  with. 

Eleven  of  the  25  subjects  interviewed  said  that  some 
kind  of  course  addressing  risk  assessments  in  the  systems 
acquisition  environment  would  be  useful.  Some  would  like  to 
see  specific  courses  set  up  to  educate  systems  acquisition 
personnel  on  risk  analysis  and  assessments.  Others  would 
like  to  see  the  addition  of  risk  analysis  to  the  basic 
systems  acquistion  formal  courses  so  that  they  may  be 
educated  as  to  where  to  look  for  more  specific  guidance. 

None  of  the  interview  subjects  were  familiar  with  the 
Defense  Systems  Management  College  manuals  on  Risk 
Assessment  Techniques  ^r  Risk  Management  Concepts,  but  most 
of  the  subjects  indicated  an  interest  in  obtaining  copies 
after  learning  of  their  existence  and  contents. 
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Conclusions  based  on  the  results  analyzed  and  discussed 
in  this  chapter  are  brought  forth  in  Chapter  V.  Along  with 
these  conclusions  are  recommendations  for  addressing  the 
problems  brought  forth  in  this  research  and  recommendations 
for  further  research  to  supplement  this  study  and  provide 
further  insight  into  the  role  of  risk  in  the  Air  Force 
weapon  system  acquisition  process. 


V.  Conclusions  and  Recommendations 

Introduction 

This  research  project  was  primarily  exploratory  in 
nature.  The  data  collected  through  personal  interviews  was 
studied  to  determine  if  generalizations  could  be  made  about 
the  "typical"  Aeronautical  Systems  Division  program  manager 
and  his  or  her  view  of  risk  and  its  relative  importance  in 
the  management  of  Air  Force  acquisition  programs.  Possible 
correlations  between  individual  program  manager  backgrounds 
and  their  views  on  the  concept  of  risk  were  studied.  The 
results  of  this  study  should  serve  as  a  starting  point  to 
guide  further  research  in  the  area  of  ris.k  in  Air  Force 
program  management. 

CQnclusiQJis 

The  objective  of  this  study  was  to  determine  how  and  to 
what  degree  Air  Force  program  managers  at  the  Aeronautical 
Systems  Division  evaluated  the  risk  associated  with  the 
various  program  decision  areas  they  faced.  These  decision 
areas  included  the  follow'ng: 

1.  Contractor  Source  Selection. 

2.  Preparation  of  the  Acquisition  Strategy. 

3.  Preparation  of  the  Acquisition  Plan. 

4.  Determination  of  the  Contract  Type. 

5.  Determination  of  Test  and  Evaluation  Objectives. 

6.  Other  Program  Specific  Areas. 
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To  meet  that  objective,  the  following  five 
investigative  questions  had  to  be  answered. 

1.  How  do  program  managers  view  the  concept  of  risk? 

2.  How  much  risk  do  they  feel  is  present  in  each  of 
the  program  management  decision  areas  outlined  above,  and 
what  kinds  of  risks  are  they? 

3.  What  means  do  program  managers  use  to  evaluate 
(formally  or  informally,  quantitatively  or  qualitatively) 
risk  in  each  area? 

4.  What  circumstances  lead  them  to  evaluate  i isk  in 
these  ways? 

5.  Do  correlations  exist  between  risk  taking 
tendencies  and  the  following  independent  variables: 

a.  Academic  Degree? 

b.  Types  of  Work  Experience? 

c.  Age? 

d.  Marital  Status? 
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indeed  true  in  the  responses  of  the  program  managers  at  ASD 
(27:1407).  The  popularity  of  the  "gut  feel”  response  was 
strong  evidence  of  this.  Few  program  managers  were  able  to 
provide  a  precise  definition  of  risk,  and  many  were  only 
able  to  describe  risky  situations  and  causes  of  risk. 

The  ASD  program  managers  in  this  study  did  not  view  the 
risk  assessments  being  performed  at  ASD  as  being  extremely 
thorough,  nor  did  they  view  them  as  being  simply  cursory. 

The  general  feeling  was  that  a  less  than  absolutely  thorough 
risk  assessment  was  adequate  in  most  cases. 

Generally,  ASD  program  managers  viewed  themselves  as 
being  risk  takers,  and  their  responses  reflected  the  feeling 
brought  forth  in  the  literature.  That  feeling  was  that  risk 
was  essential  in  order  to  be  successful  at  work.  The 
correlation  analysis  showed  no  strong  evidence  of  a 
relationship  between  describing  oneself  as  a  risk  taker  and 
possessing  a  technical  degree,  being  married,  etc. 

Finally,  the  program  managers  generally  felt  that  they 
could  use  some  additional  knowledge  in  the  areas  of  risk  and 
risk  assessments  or  analyses.  This  would  make  them  more 
effective  as  risk  managers  and  in  the  overall  management  of 
their  programs. 

Overall,  program  managers  viewed  risk  and  risk 
assessments  as  being  imprecise,  but  necessary.  There  was  a 
general  feeling  that  additional  knowledge  in  the  areas  of 
risk  and  risk  assessments  would  be  useful. 


Investigative  Question  2: _ Risk  in  Program  Decisions. 

Three  basic  types  of  risk  are  usually  addressed  in  Air  Force 
program  management.  Those  risk  types  are  cost  risk, 
schedule  risk,  and  technical  performance  risk.  Reflecting 
what  the  literature  suggested,  the  risk  types  were  not  found 
to  be  independent  of  each  other  by  the  ASD  program  managers. 
In  other  words,  program  decisions  involving  risk  affected 
the  perceived  risk  associated  with  the  cost,  schedule,  and 
technical  performance  aspects  of  the  program.  Generally, 
all  risk  types  for  each  program  management  decision  area 
were  viewed  to  be  at  least  "somewhat  important."  No  risk 
was  consistently  rated  high  or  low  across  any  particular 
program  management  decision  area  or  for  any  risk  type. 

There  were,  however,  differences  in  which  types  of  risk  were 
emphasized  in  various  program  management  decision  areas. 

For  example,  the  technical  risk  associated  with  decisions 
affecting  Test  and  Evaluation  was  much  higher  than  the  cost 
risk  associated  with  T&E . 

Overall,  program  managers  at  ASD  see  all  types  of  risk 
involved  at  each  stage  of  the  acquisition  cycle  in  the  many 
different  decisions  that  must  be  made.  The  data  indicate 
that  the  level  and  types  of  risks  emphasized  depends  on  the 
development  stage  in  which  a  program  lies  (planning, 
contract  preparation,  test,  etc.).  Also,  the  data  suggest 
that  program  managers  may  tend  to  make  concessions  in  one 
risk  type  so  that  risks  of  another  type  may  be  addressed 
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more  thoroughly.  Again,  this  seems  to  be  program  phase 
dependent . 

Investigative  Question  3:  Methods  of  Risk  Assessment. 
This  study  investigated  the  means  program  managers  used  to 
evaluate  the  risks  associated  with  their  programs.  Biromial 
tests  of  the  data  showed  that  ASD  program  managers  tended 
toward  performing  informal,  qualitative  risk  assessments 
rather  than  formal,  quantitative  risk  assessments.  Again, 
the  popularity  of  the  "gut  feel"  risk  assessment  supports 
this  conclusion.  In  addition,  the  data  suggested  that  there 
are  very  few  explicit  methods  or  techniques  of  risk 
assessment  and  analysis  being  used  by  these  program 
managers . 

Invsstigative  Question  4j _ Egaacrts..  f.Qr  -Pe.rfQxmiDg  Risii 

Assessments  in  a  Particular  Wav.  Program  managers  performed 
risk  assessments  in  the  manners  they  did  because  of  a  number 
of  reasons.  The  most  popular  reason  was  the  lack  of  time 
necessary  for  performing  more  extensive  assessments. 
Surprisingly,  only  one  of  the  25  interview  subjects 
mentioned  that  risk  assessments  were  done  in  a  particular 
manner  because  of  direction  from  superiors.  This  would 
suggest  that  the  supervisors  of  these  program  managers  may 
not  be  familiar  w:’th  risk  or  risk  assessment  methods.  No 
generalizations  or  conclusions  could  be  drawn  from  the  data 
concerning  circumstances  leading  to  performing  a  particular 
type  of  risk  assessment. 
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correlation  analysis  of  the  data  suggested  a  few 
relationships  that  could  be  significant.  The  largest 
correlation  was  found  to  exist  between  the  possession  of  a 
technical  degree  and  the  cost  risk  associated  with  Test  and 
Evaluation  (r  -  -Q.470).  This  inverse  relationship 
indicated  that  technical  degree  holders  tended  to  have  a 
lower  rating  of  cost  risk  for  T&E .  This  suggests  that 
technical  degree  holders  may  be  more  likely  to  sacrifice 
cost  risks  to  address  the  technical  and  schedule  aspects  of 
Test  and  Evaluation.  Non-techn ical  degree  holding  managers 
were  relatively  more  likely  to  favor  the  cost  risk 
associated  with  Test  and  Evaluation. 

Another  inverse  relationship  was  found  to  exist  between 
possessing  a  technical  degree  and  the  technical  risk 
associated  with  Source  Selection  (r  =  -0.442).  Those 
program  managers  with  a  technical  degree  tended  to  indicate 
lower  risk  ratings.  When  combined  with  the  above 
relationship,  the  data  suggest  that  technical  degree  holders 
may  tend  toward  sacrificing  technical  risk  early  in  the 
program  and  emphasize  it  later  in  the  program  during  the 
testing  phase. 

Two  other  possible  direct  relationships  were  found  to 
exist.  The  first  of  these  was  between  having  had  some 
operational  experience  and  the  cost  risk  associated  with 
test  and  evaluation.  Those  program  managers  with  prior 
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experience  in  operational  jobs  as  discussed  in  Chapter  I, 
were  more  likely  to  rate  the  cost  risk  associated  with  T&E 
higher  than  those  without  operational  experience. 

The  other  relationship  was  that  of  being  over  30  years 
of  age  and  the  cost  risk  associated  with  determining  the 
contract  type.  The  older  program  managers  tended  to  rate 
that  cost  risk  higher  than  the  younger  program  managers. 

The  reason  for  this  relationship  may  be  the  fact  that  the 
Air  Force  generally  applies  more  responsibility  with 
increasing  age.  Hence,  older  program  managers  may  tend  to 
be  more  concerned  with  risk  in  this  area  because  of  that 
increased  responsibility. 

Overall,  the  push  by  senior  government  and  Department 
of  Defense  officials  to  emphasize  the  importance  of  risk  and 
risk  assessments  in  the  management  of  Air  Force  acquisition 
programs  is  not  being  felt  by  these  mid-level  acquisition 
managers  at  ASD .  There  was  no  evidence  to  indicate  that  any 
standardized  risk  assessment  methods  were  in  existence  at 
ASD.  There  was,  however,  a  general  feeling  that  more 
education  was  necessary  in  the  areas  of  risk  and  risk 
assessment  and  that  more  standardization  could  occur  if  some 
guidance  were  available.  For  the  most  part,  what  was  known 
at  ASD  regarding  risk  assessments  was  learned  on  the  job  and 
not  through  any  formal  training. 


The  program  managers  at  ASD  were  generally  familiar 
with  the  quantitative  tools  necessary  to  perform  a 
quantitative  risk  assessment,  but  they  were  not  familiar 
with  how  to  apply  those  tools  toward  a  quantitative  or 
formal  assessment. 


The  fact  that  none  of  the  program  managers  interviewed 
were  familiar  with  the  DSMC  guides  on  risk  management  and 
risk  assessment  techniques  seems  to  provide  support  that 
more  education  is  necessary  in  the  areas  of  risk  management, 
risk  analysis,  and  risk  assessment.  The  program  managers  at 
ASD  supported  the  idea  of  introducing  some  type  of  risk 
education  through  professional  continuing  education  courses 
offered  at  the  Air  Force  Institute  of  Technology's  (AFIT) 
School  of  Systems  and  Logistics. 

Introduction  of  a  new  course  or  the  modification  of 
existing  courses  to  include  risk  would  aid  in  reducing  the 
inconsistencies  found  in  various  risk  assessments  at  ASD. 

It  would  aid  in  defining  a  more  concrete  role  for  risk  in 
the  systems  acquisition  arena  and  in  making  risk  assessments 
a  more  useful  management  tool  in  these  times  of  high 
visibility  programs  having  to  deal  with  tight  budgets,  short 
suspenses,  and  highly  complex  technologies. 


expansion 


this  study  to  include  program  managers  at  other  Air  Force 
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Systems  Command  product  divisions  would  help  determine 
whether  or  not  the  introduction  of  a  course  on  risk  in  the 
Air  Force  systems  acquisition  process  would  be  beneficial. 
Also,  a  study  of  how  senior  Air  Force  officials  view  the 
concept  of  risk  would  be  useful  as  a  comparison  to  how  the 
junior  and  mid-level  acquisition  managers  view  risk. 

Further  analysis  of  the  relationships  between  technical 
degree  holders  and  their  risk  taking  tendencies  could 
provide  useful  insight  into  being  able  to  predict  where  the 
risk  emphasis  will  likely  be  placed  on  a  program.  The 
analysis  of  the  data  showed  that  differences  exist  between 
technical  and  non- techn ical  degree  holding  program  managers. 
A  slight  bias  toward  risk  taking  by  technical  degreed 
managers  was  identified,  and  it  may  be  that  technical  people 
do  not  care  as  much  about  cost  or  that  non-technical  people 
do  not  understand  the  technical  risks  as  well  as  technical 
people.  It  is  important  to  know  why  this  difference 
occurred.  The  management  of  inherently  risky  programs  will 
likely  be  affected  by  these  biases  if  they  are  confirmed  to 
exist  by  further  study.  Without  a  more  in-depth 
understanding  of  this  relationship,  progress  in  managing 
program  risk  is  not  likely  to  occur. 

Finally,  a  pre-test  could  be  performed  on  students 
entering  courses  at  AFIT  or  DSMC  to  evaluate  their  knowledge 
of  risk  and  risk  assessments.  These  results  could  be  used 
to  confirm  the  results  of  this  preliminary  study. 
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Summary 


The  risks  associated  with  the  procurement  of  Air  Force 
weapon  systems  are  real  and  plentiful.  They  are  related  to 
such  problems  as  cost  overruns,  schedule  slippages,  and 
technical  roadblocks.  If  these  risks  are  investigated, 
identified,  and  properly  managed,  many  of  these  problems  can 
be  avoided  through  diversion  of  resources  and  other  means. 

In  order  to  properly  investigate,  identify,  and  manage  risk, 
program  managers  must  be  properly  educated  in  the  areas  of 
risk  assessment  and  analysis.  The  program  managers  in  this 
study  do  not  appear  to  be  properly  trained,  and  problems 
associated  with  improper  management  of  risk  could  emerge. 

The  introduction  of  some  type  of  formalized  education  at  the 
Air  Force  Institute  of  Technology  in  the  areas  of  risk 
management,  risk  assessment,  and  risk  analysis  would  be  a 
wise  move  toward  preventing  problems  relating  to  this  lack 
of  knowledge. 
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Appendix  A 


Risk  in  Decision  Making  Interview  Guide 


Interview  Control  Number 
Personal  Data 

Rank  or  Grade: _ 

Age : 

_  21-25  yrs 

_  26-30  yrs 

_  31-35  yrs 

_  36-40  yrs 

_  over  40  yrs 

Sex :  _ 


Marital  Status: 

_  Single 

_  Harried 


Number  of  Children: 


Educational  History: 

Undergraduate  Major  Field(s) 

of  Study/Degree,  Year  _ 

Completed:  _ 


Graduate  Field(s)  of  Study/ 
Degree,  Year  Completed:  _ 


Air  Force  Institute  of  Technology  Prof  ^  ^  s  X  on  Cl  1  Continuing 
Education  Courses:  _ 


Professional  Military  Education: 

_  SOS  Method 

_  ACSC  (or  equivalent)  Method 

_  AWC  (or  equivalent)  Method 
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Work  History 


Present : 

Time  Period: _ 

Location : _ 

Job  Title; _ 

Responsibilities: 


Previous : 

Ti’ne  Period;  _ 

Location:  _ 

Job  Title:  _ 

Responsibilities: 
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Risk  Definition: 

1.  How  would  you  describe  the  concept  of  risk? 


Risk  in  Decision  Making: 

2.  Describe  how  you  usually  evaluate  risk.  What  process  or 
steps  do  you  go  through?  Can  you  give  an  example? 
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3.  Do  you  use  any  other  methods  to  evaluate  risk?  If  so, 
which  ones? 


4.  Where  did  you  learn  the  methods  described  above? 


5.  Are  you  familiar  with  any  of  the  following  quantitative 
risk  assessment  tools? 

1.  Expected  Value/Utility  Analysis  _ 

2.  Network  Analysis  (PERT,  VERT,  CPM )  _ 

3.  Decision  Tree  Analysis  _ 

4.  Life  Cycle  Cost  Models  _ 

5.  Computer  Simulation  (SLAM,  SIMPLE)  _ 

6.  Have  you  been  involved  with  a  risk  assessment  in  making 
decisions  affecting  any  of  the  following  areas? 

1.  Contractor  Source  Selection:  _ 

2.  Preparing  an  Acquisition  Strategy  (Program  Management 

Plan):  _ 

3.  Preparing  an  Acquisition  Plan  risk  assessment  for  the 

Procuring  Contracting  Officer:  _ 
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4. 

5. 

6. 


Determining 
Determining 
Other  areas 


the  type  of  contract  to  be  used: 
Test  &  Evaluation  objectives:  _ 


7.  Please  indicate  in  the  matrix  below  what  types  of  risk 
(cost,  schedule,  performance)  were  associated  with  each  of 
the  above  program  decision  areas? 

In  the  same  boxes,  please  indicate  the  relative  importance 
of  each  of  these  risk  types  using  the  scale  below  (1  through 
5)? 

Hot  all  Llttla  Somawhat  Varv  Ex^raanalv 

Important  Xmportanca  Important  Important  Important 


1 


2 


3 


4 


5 


Cost  !  Schedule 

1 

1 

Source  Selection 

1 

1 

1 

» 

1 

Acquisition  Strategy  (PMP) 

1 

1 

1 

1 

1 

Acquisition  Plan 

1 

1 

1 

1 

1 

Contract  Type 

1 

1 

1 

1 

1 

T&E  Objectives 

1 

1 

1 

1 

t 

Other 

1 

t 

1 

« 

1 

1 

1 

1 

1 

1 

1 
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8.  If  risk  assessments  were  done  in  any  of  the  above  areas, 
what  methods  were  used? 


9.  Generally,  on  a  scale  from  0  to  100,  how  thoroughly  are 
risk  assessments  or  analyses  performed? 

Min  Max 

0  — -10 20 30 40 50 60 70 80 90--- 100 


10.  What  is  the  primary  reason  you  perform  risk  assessments 
the  way  you  do? 


11.  How  would  you  describe  yourself  as  far  as  risk  is 
concerned?  Risk  taker?  Risk  neutral?  Risk  avoiding?  Why? 


12.  Are  you  comfortable  with  vour  knowledge  of  risk  and 
risk  assessments?  If  not,  in  which  areas  would  you  like  to 
increase  your  knowledge? 


13.  Are  you  familiar  ’.:ith  any  DoD  guidance  on  risk 
assessment,  analysis,  and  management?  If  so,  which 
guidance?  Where  would  you  like  to  see  more  guidance? 
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Appendix  B; 


Question  1.  How  would  you  describe  the  concept  of  risk? 

1.  Risk  is  the  likelihood  of  an  unplanned  event 
happening.  The  event  has  some  sort  of  adverse  effect.  Risk 
is  not  necessarily  quantifiable,  and  it  is  a  broad  (general) 
term . 

2.  Risk  contains  areas  of  design  that  are  not 
understood  or  identified  by  both  government  and  contractor. 
It  causes  schedule  slips  and  cost  overruns.  Risk  is 
involved  with  state-of-the-art  versus  known  concepts  and 
designs.  Risk  occurs  when  the  time  period  allowed  to 
perforin  the  development  is  not  realistic  (too  short). 

3.  Generally,  risk  is  the  level  of  uncertainty  (number 
of  uncertainties).  It  asks,  "What  are  the  unknowns?"  On 
the  job,  risk  is  involved  with  high  technology.  The  money 
involved  is  not  leally  important.  All  aspects  of  risk  seem 
to  be  interrelated  (cost,  schedule,  and  performance).  It's 
a  struggle  to  quantify  risk.  All  aspects  of  risk  should  be 
tied  together  and  not  broken  out  separately  because  they  all 
have  effects  on  each  other. 

4.  Risk  is  the  degree  at  which  you'll  expect  to 
accomplish  something  within  constraints  (money,  time, 
resources,  technology).  Risk  is  derived  from  uncertainty. 
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In  other  words,  any  risk  involves  uncertainty.  At  work,  the 
degree  of  consequence  is  the  driver. 

5.  Risk  is  the  probability  that  a  program  will  fall 
into  line.  It's  made  up  of  cost,  schedule  and  technical 
risks.  Each  has  an  influence  on  the  others.  Risk  is  what 
the  individual  or  group  arrives  at.  No  specific  definition 
exists  (no  standard  rating  system). 

6.  Risk  is  the  chance  of  failure  or  loss  associated 
with  some  action  or  decision.  It's  directly  related  with 
obtaining  accurate  information  about  an  action  or  decision. 

7.  Risk  is  the  term  used  to  describe  uncertainty.  If 
one  takes  a  risk,  there  is  a  chance  that  whatever  is 
attempted  may  fail.  This  failure  could  result  in  loss  of 
money  or  possibly  injury/death  to  the  one  attempting  the 
task.  If  one  applies  this  concept  to  program  management, 
then  risk  usually  refers  to  the  possibility  of  program 
failure.  It's  useful  for  upper  management. 

8.  In  reference  to  program  management,  risk  is  the 
probability  of  success  vs  failure  of  completing  an  entire 
project  (and  portions  of  that  project)  by  meeting  the 
requirements,  schedule,  costs,  milestones,  and  goals  of  that 
project.  It's  the  likelihood  of  accomplishing  tasks  given 
cost  and  time  constraints,  limitations  of  operations 
flexibility,  or  any  other  externally  applied  constraints 
that  impact  how  or  when  management  techniques  are  applied. 
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Legislated  limitations  and  constraints  drive  time  risks. 

For  example,  sole  source  limitations. 

9.  Risk  is  something  that  impacts  the  schedule.  These 
may  be  assets,  resources,  or  money.  Consequences  aren't  the 
big  concern;  they  can  be  dealt  with  more  easily.  (As 
opposed  to  the  likelihood  of  an  event  occurring) 

10.  Risk  is  when  you  expose  yourself  to  some  type  of 
danger.  It  relates  to  output  vs  input.  Don't  expend  more 
time  and  effort  than  you'll  recover.  Risk  is  involved  with 
new  areas  of  development  rather  than  with  off-the-shelf 
hardware . 

11.  Risk  is  the  state  of  balancing  what  can  go  right 
(reward)  vs  what  can  go  wrong  (price  you  pay).  The  level  of 
risk  depends  on  the  severity  of  the  consequence.  Loss  of 
life  is  a  high  risk  situation. 

12.  The  risk  of  a  program  depends  on  the  technical 
aspects  of  the  program,  the  dollars  spent,  and  the  schedule 
impacts  of  certain  decisions.  Risk  is  subjective.  The  user 
of  the  weapon  system  drives  the  requirements  and  the 
subsequent  risks. 

13.  Risk  is  the  probability  that  I  will  achieve  a 
goal,  given  a  number  of  paths  to  choose  from  to  meet  that 
goal.  Factors  in  a  risk  assessment  include  schedule, 
budget,  technology,  and  politics. 

14.  Risk  is  the  probability  of  a  mishap  occurring. 
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15.  The  concept  of  risk  is  the  taking  of  reasonable 
chances  to  achieve  success  in  a  given  venture. 

16.  Risk  is  a  measure  of  uncertainty  applied  to  a 
certain  situation.  In  the  Air  Force  there  are  cost, 
schedule  and  technical  risks. 

17.  Risk  i*!  a  condition  in  which  failure  to  address 
issues  results  in  unacceptable  consequences  or  programmatic 
concerns.  It  mainly  deals  with  cost  and  schedule.  Risk 
always  exists.  Risk  is  different  from  uncertainty  in  that 
risk  is  quantifiable  and  deals  with  probabilities  while 
uncertainty  does  not. 

18.  Risk  is  the  difference  between  the  maximum  amount 
a  program  could  cost  if  all  reasonably  possible 
contingencies  occur,  and  the  minimum  amount  a  program  could 
"reasonably"  cost  if  a  contractor  manages  well,  is 
innovative,  and  does  proper  planning.  Technical  and 
schedule  risks  evolve  into  cost  risks. 

19.  Risk  is  the  uncertainty  induced  by  schedule  or 
budget  problems  caused  by  unrealistic  expectations.  Risk 
arises  when  proposed  schedules  are  unrealistic.  Our  risks 
are  generally  low  because  we  don't  really  push  the  state-of- 
the-art  . 

20.  Risk  is  the  possibility  of  not  reaching  a  goal 
technologically,  manager ially ,  or  financially. 

21.  Risk  occurs  when  you  choose  a  course  of  action 
which  allows  the  opportunity  for  failure. 
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22.  Risk  is  a  measure  of  the  susceptibility  to 
failure . 

23.  Risk  is  the  probability  of  occurrence,  given 
possible  outcomes.  Risk  analysis  involves  defining  the 
risk,  consciously  looking  at  it,  and  separating  it  from 
uncertainty . 

24.  Risk  involves  putting  yourself  in  a  position  to 
where  you  are  in  a  gambling  situation.  Factors  include 
schedules  and  technology. 

25.  Risk  is  the  impact  of  a  decision  to  be  made.  In 
the  Air  Force,  this  involves  schedule  decisions,  money 
decisions,  and  technical  performance  decisions.  RisK 
assessment  is  a  day-to-day  iterative  process. 

Question  2.  Describe  how  you  usually  evaluate  risk.  What 
process  or  steps  do  you  go  through?  Can  you  give  an 
example? 

1.  I  don't  use  any  quantitative  method  of  evaluating 
risk.  I  use  a  subjective  approach.  The  size  of  the  program 
dictates  the  level  of  assessment.  My  program  is  a  small 
program  using  lots  of  pre-developed  items,  i.e.  no  risk. 

Any  assessments  I  do  are  inform*’!,  based  on  past  experience. 

2.  First,  I  try  to  understand  the  design  concepts 
and/or  problems  on  a  detailed  level  up  front.  Then  I 
perform  early  simulations  and  studies  to  define  the  concept 
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and  logic  of  the  design,  and  I  update  these  throughout  the 
developjient  to  reflect  the  current  design.  I  then  get 
involved  with  the  testing  of  the  subsystem/system  at  the 
earlie'  point  and  continue  through  system  maturity. 

Fi''  ily,  I'll  perform  an  independent  ver  i  f  icat  ion/ val  idat  ion 
of  the  design  using  consultants  or  Air  Force  experts.  I 
feel  that  the  ratings  of  "high,"  "low,"  and  "moderate"  are 
insufficient  to  adequately  categorize  risk.  They  are  just 
too  broad  and  vague. 

3.  First,  what's  the  bottom  line?  What  technologies 
am  I  dealing  with,  and  what  is  the  status  of  the  effort?  In 
evaluating  risk,  I  rely  primarily  on  prior  experience,  and 
my  knowledge  of  the  system.  I  also  rely  on  expert 
experience  for  technical  matters.  General  lessons  learned 
are  useless.  Specific  lessons  learned  are  much  more 
helpful.  I  usually  end  up  telling  people  what  they  want  to 
hear  as  far  as  low,  moderate,  or  high  risk  is  concerned. 
Political  influences  are  very  strong  and  can  cause  a 
legitimate  risk  assessment  to  be  "thrown  out  the  window"  so 
that  a  more  favorable  "assessment"  can  be  briefed  up  the 
management  chain. 

4.  My  risk  assessments  have  no  real  pattern.  I  don't 
do  any  formal  risk  assessments.  When  I  assess  the  risk  of  a 
situation  I  collect  all  the  information  I  can  and  get  a  "gut 
feel"  for  the  risk.  I  obtain  that  information  by  consulting 
experts  in  the  particular  fields  and  by  calling  on  my  own 
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experiences.  I  get  a  feel  for  risk  by  thinking  through 
causes  and  effects. 

5.  T  assess  risk  using  a  "gut  feel"  based  primarily  on 
past  experience.  I  refine  that  assessment  by  using  expert 
opinions.  As  far  as  cost  risk  is  concerned,  I  address  trade 
studies  to  see  what  I  can  get  with  a  given  budget.  I  then 
assess  how  acceptable  that  is.  Generally,  I'll  look  at  risk 
up  front  in  the  program  and  then  make  spot  checks  through 
the  life  of  the  program.  I  do  this  because  the  risks  of  a 
program  are  highly  variable. 

6.  In  my  job  I  evaluate  risk  mainly  through  a  common 
sense  approach.  I  simply  weigh  the  pluses  and  minuses  in  my 
head.  I  don't  really  have  a  formalized  process  with  steps 
that  I  go  through. 

7.  Risk  is  evaluated  in  a  subjective  manner  with 
certain  objective  inputs.  For  instance,  when  evaluating 
program  risk,  one  must  consider  funding,  schedule, 
complexity  of  tasks,  new  technology,  and  program  personnel. 
Most  of  these  can  be  measured  objectively,  but  the 
combination  of  all  of  them  is  usually  too  complex  to  be  able 
to  measure  with  any  certainty.  A  program  pushing  advanced 
technology  and  underfunded  would  appear  to  have  a  large 
degree  of  risk,  yet  with  a  technology  breakthrough  and 
congressional  interest,  the  program  could  end  up  quite 
successful.  Information  is  collected  using  contractor  past 
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performance  data  and  is  gathered  from  experts  on  the 
government  side. 

8.  First,  you  gather  information:  background, 
requirements  from  the  user,  tasks  required,  funding 
constraints/limitations,  historical/past  performance  record, 
reliability/credibility,  technological  rating  similar 
program  precedence  and  performance,  etc.  Second,  you 
assemble  the  information  into  a  coherent  structure: 
costs/schedule/problems/constraints/barriers .  Then  you 
define  and  evaluate  the  critical  path  to  completion.  You 
evaluate  each  subtask  and  barrier.  You  then  estimate  the 
probabilities  for  each  successful  subtask  completion. 
Finally,  you  evaluate  the  overall  probability  for  project 
completion . 

9.  I  look  at  resources  first  (hardware,  people,  and 
money).  Then  I  look  at  the  near  term  situation  and  evaluate 
what  possible  impacts  there  are  to  the  schedule.  Schedule 
is  the  primary  risk  associated  with  a  program  because  of 
user  needs.  We  need  to  meet  a  specified  Initial  Operatic,  -j. 
Capability  date.  You  may  be  forced  to  jeopardi;’e  technical 
performance  to  meet  schedule  requirements. 

10.  I  develop  alternatives  to  minimize  risk.  What  s 
the  backup  plan?  I  usually  use  a  gut  feel  for  my  risk 
assessments  because  of  time  consxderations .  I'll  bounce 
questions  off  experts  to  avoid  "making  the  same  mistake 
twice  .  " 
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11.  I'll  ask  experts  on  what  is  good  or  what  is  bad 
with  the  program.  I  try  to  get  several  opinions  and  then 
make  my  own  assessment  based  on  all  of  the  input. 

12.  Past  history  is  very  important.  I'll  look  at  my 
program's  similarity  to  other  programs.  I  gather 
information  from  the  various  project  management  teams  and 
support  people.  I  then  pool  the  information  gathered  and 
make  a  recommendation  of  my  own.  Software  is  a  big  risk. 

It  requires  much  more  attention  because  of  all  of  the 
unknowns  associated  with  software  development. 

13.  First,  define  your  problem.  How  important  is  it 
for  the  "all  around"  program  or  the  "big  picture?"  Second, 
we  make  a  committee  decision.  I  gather  the  experts  together 
and  we  make  a  decision  based  on  inputs  from  the  functionals. 

14.  My  risk  assessments  are  based  on  prior  experience. 
Precedence  is  the  most  important  aspect.  Working  as  a  wing 
safety  officer,  we  worried  whether  or  not  we  were  going  to 
kill  somebody,  and  we  made  risk  assessments  based  on  similar 
missions  in  the  past.  In  the  acquisition  area  I  rate  risks 
high,  medium,  and  low.  I  rate  something  a  high  risk  when  I 
know  "something  will  eat  me  up."  I  rate  it  medium  when 
there  is  a  50-50  chance  of  something  happening,  and  I  rate 
something  low  risk  if  I  feel  there  is  an  80%  probability 
that  the  program  or  decision  will  be  a  success. 

15.  I  usually  assess  the  situation  by  listing  the  pros 
and  cons  and  then  establishing  the  most  likely  alternative 


99 


for  success  (given  a  minimum  of  unknowns).  It's  a  mental 
process  that  produces  a  gut  feel  based  on  past  experience. 

16.  My  risk  assessments  are  based  on  intuition  or  gut 
feel.  Historical  programs  help,  and  I'll  talk  to  experts  in 
any  specific  areas  that  I  am  not  familiar  with. 

17.  I  don't  use  any  quantitative  methods  for 
evaluating  risk.  My  risk  assessments  are  usually  informal, 
based  on  personal  experience.  I'll  tap  into  the  laboratory 
for  their  opinions  on  highly  technical  areas  of  the  program, 
and  I'll  use  data  from  other  programs  to  possibly  learn  from 
their  mistakes.  Our  program  attempts  to  minimize  risk  by 
conducting  periodic  design  reviews  and  production  readiness 
reviews . 

18.  We  try  to  predict  which  events  in  the  contractor's 

estimates  are  bound  to  occur.  We  try  to  separate  out  the 
events  we  are  not  so  sure  about  occurring.  This  provides 
the  basis  for  our  low  to  high  risk  items,  depending  on  how 
sure  we  feel  about  the  probability  of  success.  These 
predictions  are  based  on  input  from  the  experts  in  the 
field:  engineers,  contracting  officers,  logisticians,  etc. 

19.  My  risk  evaluation  process  is  a  data  gathering 
process.  I  do  some  background  research,  and  I'll  talk  to 
people  who  work  in  specific  areas.  My  risk  assessment  is 
then  based  on  my  gut  feel  for  the  situation. 

20.  I'll  evaluate  the  approach  we ' re  taking  and  use 
historical  data.  I'll  also  check  with  other  people.  It's 


really  usually  an  informal  process  and  ends  up  being  a  gut 
feel  based  on  experience. 

21.  First,  look  at  what  the  issue  is.  Look  at  past 
performances.  Ask,  what's  the  consequence  of  the  contractor 
not  getting  something  right?  What  can  be  done  to  work 
around  it?  I'll  rely  on  information  furnished  by  the 
contractor,  information  provided  by  tech  people  and 
engineers,  and  information  found  in  past  performances.  My 
risk  assessments  have  all  been  informal  risk  assessments. 

22.  I  evaluate  risk  based  on  past  experience.  Risk  is 
evaluated  by  determining  what  the  results  of  failure  are. 
Management  tools  (schedules  &  budgets)  are  also  widely  used 
to  evaluate  risk.  Again,  evaluations  are  based  on  intuition 
and  experience. 

23.  I  have  no  “canned"  process  for  evaluating  risk. 

We  usually  evaluate  risk  with  the  "seat  of  our  pants" 
approach.  A  weakness  is  that  we  don't  always  look  at  the 
threat  and  overall  mission  of  the  Air  Force  when  evaluating 
risk.  I  will  rate  risk  on  a  high/moderate/ low  scale  which 
is  subjective  but  based  on  experience  and  job  expertise. 

The  risk  assessments  are  only  as  accurate  as  the  information 
that  is  inputed . 

24.  It's  an  informal  evaluation.  I  run  through  folks 
who've  done  the  job  before.  I  look  at  contractor 
experience.  I  go  to  technical  experts  for  information.  I 
also  look  at  historical/previous  work  in  the  area. 
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Experience  is  very  valuable.  Because  of  the  nature  of  my 
program,  I  haven't  dealt  with  any  high  risk  situations,  and 
any  risk  assessments  were  strictly  informal. 

25.  I  rely  on  a  team  of  functionals  to  provide  me 
information  on  my  risk  assessments.  I  can't  afford  to  get 
down  into  the  nuts  and  bolts,  so  I  rely  heavily  on  my 
support  people.  Risk  is  a  day-to-day  thing,  and  the  risk 
evaluation  of  any  situation  needs  to  be  updated  regularly. 

I  make  use  out  of  historical  data  whenever  it  exists,  but 
with  many  Air  Force  systems,  a  historical  precedent  is 
simply  not  there. 

Question  3.  Do  you  use  any  other  methods  to  evaluate  risk? 
If  so,  which  ones? 

1.  Not  really,  it's  mainly  just  a  subjective 
evaluation . 

2.  I  try  to  increase  my  own  knowledge  base  all  the 
time  so  that  I  can  become  more  confident  with  the 
assessments  I  make. 

3.  I  don't  use  any  quantitative  methods  (number 
crunching).  I  look  at  contractor  past  performance  and  make 
use  of  scheduling  tools. 

4.  I've  used  PERT  charts  for  schedule  risk  evaluation, 
and  that  was  very  helpful.  Unfortunately,  we  are  often  told 
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what  the  risk  of  the  program  will  be  by  direction  from 
superiors . 

5.  No. 

6.  No. 

7.  I  have  used  the  PERT  networking  technique. 

8.  My  risk  evaluation  is  dependent  upon  the  type  of 
project  and  program  being  run.  Risk  evaluation  should  be 
tailored  to  suit  an  individual  program  or  project.  Blending 
methods  to  suit  a  particular  program  is  best.  You  need  to 
know  where  you  feel  comfortable.  This  allows  you  to  make  a 
gut  feel  assessment  of  the  risks  involved. 

9.  PERT. 

10.  In  the  software  area,  I've  used  the  COCOhO  life 
cycle  cost  model. 

11.  I've  used  PERT  charts  and  life  cycle  cost  models. 
We  used  the  life  cycle  cost  models  because  it  was  required 
in  the  cost  estimate. 

12.  No,  not  any  others  personally.  I  have  seen  use  of 
life  cycle  cost  models  and  extensive  trade  study  analysis, 
but  because  of  time  constraints,  I  haven't  used  any  in-depth 
analysis . 

13.  I  haven't  use  any  quantitative  tools.  I  feel  that 
a  risk  assessment  is  inherent  in  any  decision.  The  size  and 
seriousness  of  the  issue  will  dictate  how  extensive  the  risk 
assessment  is. 


103 


14.  I've  used  MIL  STD  882B,  the  System  Safety  Program 
Requirements  regulation.  This  was  for  my  time  as  wing 
safety  officer. 

15.  No,  I  have  not  yet  worked  on  a  program  that  a 
major  quantitative  analysis  had  to  be  used.  However,  I 
would  willingly  employ  such  a  tool  when  warranted. 

16.  No. 

17.  I've  used  contractor  producibility  trade  studies, 
whereby  design  alternatives  are  weighed  with  respect  to 
cost,  technical  feasibility,  and  risk. 

18.  No. 

19.  We  use  CSNAS  (Computer  Supported  Network  Analysis 
System)  to  do  a  PERT  analysis  and  identify  critical  paths  in 
our  manufacturing  plan.  We  also  use  independent  review 
teams  to  review  our  program  and  determine  their  own  risk 
assessment . 

20.  I've  used  Milestone  and  Gantt  charts. 

21.  Not  personally.  I've  heard  about  the  DSMC  risk 
model  for  quantitative  risk  analysis,  but  for  our  purposes, 
the  utility  would  be  quite  low. 

22.  No. 

23.  No. 

24.  No. 

25.  I  use  common  sense.  Sometimes  I'll  look  at  the 
historical  background  of  a  program  where  lessons  learned  can 
be  helpful.  I'll  also  tap  into  the  bosses  power  to  reduce 
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the  risks  of  my  program  (additional  funds,  slipped  schedule 
requirements,  etc.).  Risks  may  be  driven  down  because  of 
vested  political  interests  from  above. 

Question  4.  Where  did  you  learn  the  methods  described 
above? 

1.  Personal  experience  and  common  sense. 

2.  From  experience,  my  own  and  others. 

3.  Experience. 

4.  On  the  job  experience.  I  learned  PERT  at  school 
and  in  my  PCE  courses. 

5.  On  the  job  trial  and  error.  Informal  training  from 
people  in  the  program  office.  Operational  experience  has 
helped  in  some  areas. 

6.  N/A 

7.  I  learned  PERT  in  a  PCE  short  course  and  in  my 
masters  courses. 

8.  Some  of  it  came  from  management  education,  some 
from  trial  and  error,  some  is  just  common  sense.  Some  came 
from  experienced  individuals,  and  some  came  from  my  own 
experiences . 

9.  From  my  contractor  and  Systems  200  course  at  AFIT. 

10.  At  school  and  at  work. 

11.  From  the  contractor  and  from  work. 
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12. 

On 

the  job 

is  where  you  learn.  Experts  let  you 

know  what  you 

do  in 

areas  unfamiliar  to  you. 

13. 

N/A 

14. 

The 

System 

Safety  Officer  course  at  Norton  AFB  and 

my 

own  p 

r  ior 

exper ie 

nee  . 

15. 

N/A 

16. 

N/A 

17. 

Schooling, 

experience,  and  technical  publications. 

18. 

Con 

tractin, 

g  courses,  the  FAR. 

19. 

For 

CSNAS, 

I  attended  an  AFALC  course. 

20. 

Exp 

er ience 

• 

21 . 

In 

the  SPO 

(program  office). 

22. 

By 

living 

it . 

23. 

Job 

exper i 

ence . 

24. 

No 

formal 

training,  mainly  experience  driven. 

25. 

On 

the  job 

experience . 

Question 

10. 

What  i 

s  the  primary  reason  you  perform  risk 

assessments  the  way 

you  do? 

1 . 

T  ime 

restr i 

ctions  is  the  primary  reason.  Because 

of 

the  n 

ature 

of  the 

program  I  work  on,  everything  being 

off 

-  the- 

shelf 

,  there 

is  very  little  risk  involved. 

2. 

You 

need  to 

be  active  in  the  design  process  and  get 

involved.  Addressing  risk  early  in  the  program  is  most 
important . 
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3.  Because  of  tasking.  Risk  assessments  are  political 
if  they  are  required.  They  are  reactionary  and  done  only 
after-the-fact  (after  a  problem  has  emerged).  The  level  of 
the  risk  assessment  is  usually  pre-determined .  Generally, 
risk  assessments  are  not  used  for  managing  programs. 

4.  Because  of  time  constraints.  Also  resource 
constraints  drive  what  type  of  risk  assessment  is  performed. 

5.  It's  what  I'm  comfortable  with,  and  it's  worked  so 

far  . 

6.  I  haven't  been  introduced  to  any  formal  techniques, 
and  I  don't  have  the  database  to  perform  the  ones  I  know. 

7.  Because  of  experience.  Number  crunching  is  not  as 
effective . 

8.  Because  of  limited  resources  and  competition  for 
funds.  I'm  skeptical  about  some  contractors.  I  don't  want 
to  get  burned . 

9.  Because  the  tools  are  not  available  and  the 
information  is  needed  yesterday. 

10.  To  reach  the  best  possible/cos :  effective/optimum 
solution  to  a  problem. 

11.  Because  of  time.  What  is  the  quickest  way? 

12.  Cost  is  the  big  driver.  I  need  the  best  way  at 
the  least  cost. 

13.  Because  it's  the  way  I've  learned  and  it's  worked 
given  the  importance  and  time  available,  but  political 
drivers  do  exist. 
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14.  I  do  risk  assessments  the  way  I  do  them  to  help  me 
direct  resources  (High  risk  requires  more  resources). 

15.  I  have  become  very  comfortable  with  my  personal 
rule  of  thumb  of  risk  caking/assessments.  This  is  based 
strictly  on  experiences  gained  through  the  years  rather  than 
structured  detailed  procedure. 

16.  Because  of  time  constraints.  Because  of  lack  of 
knowledge  of  other  methods.  Also,  "The  colonel  will  end  up 
putting  what  he  wants." 

17.  It's  whatever  comes  naturally.  Risk  assessment  is 
inherent  in  the  pragmatic  program  management  process. 

18.  We  will  do  a  thorough  risk  as=essmeut  but  wc  won't 
use  it. 

19.  It's  the  only  way  I  know  how. 

20.  I  do  them  for  peace  of  mind.  Time  is  a  driver. 

21.  I  do  them  to  answer  questions  I  have  about 
schedule,  cost,  and  performance. 

22.  Speed.  It  may  be  less  accurate,  but  it's  quick. 
Accuracy  isn't  required  in  most  cases.  The  exception  is  in 
source  selection. 

23.  Because  of  lack  of  time  and  information.  Possibly 
because  of  a  lack  of  proper  tools. 

24.  It's  an  effective  way  of  doing  it,  schedule  being 
the  driver. 

25.  We  do  them  to  satisfy  user  needs  with  schedule 
being  our  big  driver. 
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Question  11.  How  would  you  describe  yourself  as  far  as  risk 
is  concerned?  Risk  taker?  Risk  neutral?  Risk  avoiding? 
Why? 


1.  Risk  taker.  I  feel  that  I  have  good  enough 
judgment.  I  avoid  bad  risks,  i.e.  low  probabilities  of 
occurrence . 

2.  Based  on  past  failures,  I'm  risk  avoiding.  At 
work,  we're  forced  to  take  the  low  risk  approach. 

3.  I  lean  toward  being  risk  avoiding.  I  value  my  own 
security.  I'll  take  more  risks  at  work  than  with  my 
personal  life. 

4.  Personally,  risk  taker.  You  need  to  take  risk  to 
get  anywhere.  On  the  job,  risk  neutral  because  of  the 
environment . 

5.  Depends  on  the  type  of  risk.  Avoid  high  cost  risk. 
Schedule  risk  depends  on  the  urgency  of  need.  For  technical 
risk,  what  can  the  user  live  with? 

6.  Risk  taker.  Because  it's  better  to  progress  than 
to  hold  off  just  to  avoid  a  bad  decision.  Mistakes  are 
learning  experiences.  I  have  no  real  fear  of  the  political 
repercussions . 

7.  Risk  taker.  I  don't  feel  any  pressure  to  be 
anything  less.  If  they  don't  like  me,  I'll  go  back  to 
flying . 
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8.  Personal  risk  -  risk  taker.  These  are  known 
quantities.  Professionally  -  risk  averse.  I'm  dealing  with 
government  money.  I  can't  gamble  with  what  doesn't  belong 
to  me.  Uncertainties  are  what's  feared.  Known  risks  are 
acceptab le . 

9.  Risk  taker.  If  you  don't  try  something  new,  how 
will  you  ever  know  the  right  way  of  doing  anything  or 
whether  you  made  the  right  decision.  You  know  things  can  be 
accomplished  the  way  they've  been  done  before,  but  is  the 
old  method  efficient?  Can  you  save  time,  money  or 
performance  by  altering  the  status  quo? 

10.  I 'm  a  risk  taker  to  make  a  project  move  that  needs 
moving.  I'm  risk  averse  when  the  expected  outcome  is  less 
than  the  input . 

11.  Usually  risk  avoiding.  If  you  screw  up,  you  look 
and  feel  bad.  People  remember  your  screw  up.  It  really 
depends  on  the  payoff.  I'll  take  a  risk  with  a  big  program 
if  the  payoffs  are  worthy. 

12.  It's  situational.  I  take  risks  when  the  need  is 
urgent . 

13.  Personally,  not  a  risk  taker.  Professionally,  you 
need  to  take  risks.  You  have  to  take  risks  to  get  things 
done.  It's  very  much  a  function  of  the  environment. 

14.  Risk  taker.  You  do  it  yourself,  you  get  things 
done.  While  flying,  I'm  more  conservative.  I  need  to  know 


all  information . 


15.  Risk  taker.  This  is  based  on  working  in  the 
electronic  warfare  SPO  between  1964  and  1979. 

16.  Risk  neutral.  Because  I  like  positive  outcomes. 
Then  it  doesn't  hurt  as  bad  if  something  doesn't  work. 

17.  Risk  taker.  I  pretend  to  be,  but  I'm  probably  not 
as  much  as  I  think  I  am. 

18.  Risk  avoiding.  With  limited  resources,  I'd  rather 
spend  the  taxpayer's  money  on  a  sure  thing  before  I  spend  it 
on  a  very  risky  deal.  Pilots  are  high  risk  takers  by  the 
nature  of  their  business.  By  using  pilots  in  program 
management  positions  we  tend  to  buy  risk  before  a  sure 
thing . 

19.  Lean  toward  risk  taker.  I'm  usually  optimistic 
about  schedules,  etc. 

20.  Risk  taker.  You  get  benefits  from  building  and 
fixing  rather  than  "studying  it  to  death." 

21.  Risk  taker.  I  fly  and  jump  out  of  airplanes.  I 
would  tend  to  take  more  personal  risk  than  risking  the 
government . 

22.  I  will  take  on  a  risk,  then  try  to  minimize  or 
reduce  the  chances  of  failure, 

23.  I  lean  slightly  toward  risk  taker.  You  have  to 
make  decisions  without  perfect  information.  You  can't  wait 
on  others  to  make  decisions  for  you. 

24.  Slightly  risk  taking.  You  press  ahead  because 
it's  necessary.  I  don't  put  myself  or  my  program  in 
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jeopardy.  There  are  times  when  you  have  to  ?tick  your  neck 
out . 

25.  Somewhere  slightly  risk  taking.  Nothing  ventured, 
nothing  gained.  Success  is  based  on  how  often  that  works  as 
a  philosophy. 

Question  12.  Are  you  comfortable  with  your  knowledge  oj. 
risk  and  risk  assessments?  If  not,  in  which  areas  would  you 
like  to  increase  your  knowledge? 


1.  No. 

I 'd  to 

learn  more 

ab 

out  formal  risk 

assessments 

and  some 

quantitative 

methods . 

2.  No. 

I  need 

to  improve 

my 

knowledge  of  ri 

sk 

indicators  . 

I  need 

to  know  how 

to 

avoid  risk  and 

how  to 

stop  it.  Tools  are  good  only  if  they  reflect  reality. 


Yes.  Because  of  experience.  I  would  like  to  know 


more  abou 

t  t 

ools  to  be  smart  on 

other  people ' s 

as 

sessmen  ts . 

4  . 

No  . 

More  familiarity 

and  experience 

is 

necessary . 

5. 

No 

trouble  as  long  as 

a  logical  path 

is 

laid  out 

and  risk 

is 

communicated  well. 

Since  inputs  t 

o  q 

uan  t i tat i ve 

tools  are  not  common,  their  usefulness  is  not  really  clear. 
Information  being  good  is  critical. 

6.  No.  I'd  like  to  learn  more  about  the  application 
of  formal  processes  to  p.  ject  management. 
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7.  No.  Would  like  to  know  more  about  quantitative 
methods.  The  computer  can  back  you  up.  Tools  give  meat 
behind  a  subjective  assessment. 

8.  No.  You  can't  ever  know  all  there  is,  but  you  may 
want  to  know  more.  The  more  you  know,  the  better  off  you 
are.  Until  there  is  a  big  risk  you  may  not  need  any 
additional  knowledge.  Several  methods  for  doing  a  risk 
assessment  may  be  more  useful. 

9.  Not  necessarily  comfortable  with  my  gut  feel 
decisions.  More  information  may  be  useful.  Tech  experts 
are  not  always  easy  to  find,  and  tools  are  not  always 
accessible . 

10.  No.  I'd  like  to  be  more  familiar  with  some  of  the 
quantitative  tools  to  utilize  for  risk  assessment. 

11.  Don't  know.  I  don't  have  enough  knowledge  to 
really  answer  the  question.  I've  had  no  formal  instruction 
on  how  to  do  a  risk  assessment. 

12.  Yes.  I'm  as  knowledgeable  as  I  have  time  to  be. 

I  rely  on  my  functionals.  I  make  a  subjective  assessment  of 
risk  from  the  functional  inputs. 

13.  Yes.  You  learn  by  doing  mostly.  I  may  like  to 
see  some  management  schooling  in  this  area,  though. 

14.  No,  not  in  the  acquisition  environment.  I'm  just 
too  unaware  of  the  outside  forces. 

15.  I'm  comfortable  with  the  programs  I've  worked  so 
far.  I  would  like  to  increase  my  knowledge  in  the  decision 
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tree  analysis. 

16.  No.  Some  more  knowledge  of  some  of  the 
quantitative  tools  would  be  useful. 

17.  I'm  comfortable  for  the  most  part.  Don't  feel  so 
much  like  I  need  an  increase  in  knowledge,  but  to  more 
effectively  implement  some  of  the  more  quantitative, 
scientific  techniques  as  discussed. 

18.  Technical  risk  is  tough  for  me.  We  depend  on 
technical  evaluations  from  out  engineers  who  are  often  too 
busy  to  support  all  our  efforts. 

19.  No.  I  could  use  more  experience. 

20.  So  far,  what  I  know  has  been  adequate.  There  are 
some  weaknesses,  and  some  increased  knowledge  may  be 
benef icial . 

21.  Yes.  For  my  purposes,  I'm  comfortable.  If  the 
situation  changed,  possibly  not. 

22.  No.  I  want  to  get  away  from  the  gut  feel,  or  at 
least  have  another  measure  to  compare. 

23.  Yes,  but  not  totally  comfortable.  The  more  you 
know,  the  more  you  don't  know. 

24.  Yes.  It's  sufficient  for  the  moment,  but  that 
could  change  with  a  more  complex  program. 

25.  Yes,  for  the  level  I  am  at,  but  that  may  change 
later . 
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Question  13.  Are  you  familiar  with  any  DoD  guidance  on  risk 
assessment,  analysis,  or  management?  If  so,  which  guidance? 
Where  would  you  like  to  see  more  guidance? 

1.  No. 

2.  Yes,  I'm  familiar  with  a  manual  on  software  risk 
assessment.  It's  too  vague.  It  shows  how  to  identify  risk 
indicators  but  gives  no  solutions. 

3.  No.  Risk  assessment  guides  would  be  useful. 

4.  I'm  familiar  with  risk  assessment  templates,  but 
generally  they're  not  useful.  I  would  find  an  informal 
process  more  effective  and  efficient. 

5.  No.  A  risk  assessment  guide  would  be  useful  for 
further  questions  if  available. 

6.  No. 

7.  No.  It  would  be  nice  for  PCE  courses  to  cover  risk 
assessment . 

8.  No. 

9.  Other  than  what  I  learned  in  SYS  200,  no.  I  would 
like  to  see  more  guidance.  I  don't  want  to  be  like  some 
supervisors  I've  known  who  are  afraid  to  make  decisions 
because  they  don't  have  the  means  of  interpreting  the  data 
they  have.  A  course  at  AFIT  may  be  useful.  Decisions  don't 
seem  to  be  made,  and  the  ability  to  assess  risk  would  help 
make  decisions . 
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10.  No.  A  reference  manual  would  be  useful,  and  a 
course  would  be  useful. 

11.  No.  It  would  be  helpful  to  know  how  to  get 
guidance.  Risk  assessment  techniques  could  be  an  addition 
to  a  systems  acquisition  course. 

12.  No.  What  is  known  is  learned  on  the  job.  You 
need  to  trust  the  SPO  people's  knowledge.  Short  courses 
would  be  useful  to  increase  people's  ability  to  perform 
assessments . 

13.  No. 

14.  MIL  STD  882B.  Risk  should  be  introduced  to 
coursework . 

15.  No. 

16.  No. 

17.  I'm  aware  that  they  exist,  but  I  cannot  cite  the 

regulations  or  guidance.  I'm  aware  of  the  Willoughby 
Templates  for  identifying  risk  areas. 

18.  AFSCR  70-15,  Federal  Acquisition  Regulation. 

19.  No.  Could  use  it  in  acquisition  classes. 

20.  No.  Could  use  some  material  for  reference.  Would 

be  useful  to  add  into  an  acquisition  class. 

21.  DSMC  cost  models,  Willoughby  Templates. 

22.  No.  Guidance  would  be  useful  in  an  acquisition 
class . 

23.  Yes. 
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24.  Yes.  The  tech  library  has  lots  of  info.  A  macro 
introduction  into  risk  assessment  would  be  useful  so  that 
program  managers  would  know  where  to  look  for  information 
when  they  needed  it. 

25.  No.  A  top  level  school  that  doesn't  necessarily 
get  into  the  specifics  would  be  useful. 
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Appendix  C: 


STORM 

DATA 

SET 

LISTING 

DETAILED 

PROBLEM 

DATA  LISTING 

FOR 

COST  RISK 

RELATIVE 

IMPORTANCE 

ROW  LABEL 

SOURCE 

SEL 

ACQ  ST RAT 

ACQ  PLAN 

CONTRACT 

CASE 

1 

2. 

4. 

4. 

3. 

CASE 

2 

4. 

3. 

3. 

4. 

CASE 

3 

5. 

5. 

CASE 

4 

5. 

4. 

5. 

4. 

CASE 

5 

4. 

5. 

5. 

4. 

CASE 

6 

4. 

3. 

3. 

4  . 

CASE 

7 

4. 

3. 

4. 

5. 

CASE 

8 

4  . 

4. 

5. 

5. 

CASE 

9 

5. 

3. 

4  . 

4  . 

CASE 

10 

4. 

3. 

5. 

4. 

CASE 

11 

3. 

5. 

5. 

5. 

CASE 

12 

3. 

2. 

4. 

CASE 

13 

3. 

5. 

5. 

CASE 

14 

4. 

4. 

4. 

5. 

CASE 

15 

5. 

4. 

4. 

5. 

CASE 

16 

5. 

3. 

3. 

4  . 

CASE 

17 

4. 

3. 

5. 

5. 

CASE 

18 

5. 

5. 

5. 

CASE 

19 

4. 

4. 

2. 

4. 

CASE 

20 

3. 

4. 

4  . 

4  . 

CASE 

21 

4. 

5. 

5. 

5. 

CASE 

22 

5. 

3. 

3. 

4  . 

CASE 

23 

4. 

3. 

3. 

4. 

CASE 

24 

4. 

3. 

3. 

5. 

CASE 

25 

4. 

3. 

3. 

4. 

T&E 
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COCJPOOJifk  to  O>ili.O3tOrf»'CO05O5tO*i.itwC0>t>’C0O)C0tO 


STORM  DATA  SET 

LISTING 

DETAILED 

PROBLEM  DATA 

1  LISTING 

FOR 

SCHEDULE 

RISK  RELATIVE  IMPORTANCE 

ROW  LABEL 

SOURCE 

SEL 

ACQ  STRAT 

ACQ  PLAN 

CONTRACT 

CASE 

1 

3. 

5. 

4. 

1. 

CASE 

2 

3. 

3. 

3. 

4. 

CASE 

3 

, 

4. 

3. 

CASE 

4 

4. 

5. 

5. 

5. 

CASE 

5 

4. 

5. 

5. 

4. 

CASE 

6 

4. 

4. 

3. 

2. 

CASE 

7 

3. 

4. 

4. 

3. 

CASE 

8 

2. 

4. 

5. 

4. 

CASE 

9 

2. 

4. 

4  . 

3. 

CASE 

10 

5. 

5. 

5. 

3. 

CASE 

11 

4. 

5. 

5. 

3. 

CASE 

12 

3. 

4. 

3. 

4. 

CASE 

13 

, 

3. 

3. 

4. 

CASE 

14 

3. 

3. 

4. 

3. 

CASE 

15 

5. 

4. 

4. 

4. 

CASE 

16 

5. 

4. 

4. 

4. 

CASE 

17 

3. 

4. 

4. 

3. 

CASE 

18 

5. 

5. 

3. 

CASE 

19 

4. 

4. 

5. 

4  . 

CASE 

20 

3. 

4  . 

4. 

3. 

CASE 

21 

4. 

4. 

4. 

5. 

CASE 

22 

5. 

3. 

4. 

4. 

CASE 

23 

4. 

3. 

3. 

4. 

CASE 

24 

4. 

4. 

4. 

3. 

CASE 

25 

4. 

4. 

4  . 

3. 
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STORM  DATA  SET  LISTING 
DETAILED  PROBLEM  DATA  LISTING  FOR 
TECHNICAL  RISK  RELATIVE  IMPORTANCE 


ROW  LABEL 

SOURCE  SEL 

ACQ  STRAT 

ACQ  PLAN 

CONTRACT 

T&E 

CASE  1 

5. 

5. 

1. 

5. 

5. 

CASE  2 

5. 

3. 

3. 

3. 

5. 

CASE  3 

5. 

4. 

4. 

CASE  4 

4  . 

3. 

5. 

3. 

5. 

CASE  5 

5. 

4. 

4. 

5. 

5. 

CASE  6 

4. 

4. 

3. 

4. 

CASE  7 

5. 

4. 

4. 

3. 

5. 

CASE  8 

4. 

5. 

5. 

5. 

5. 

CASE  9 

4  . 

3. 

3. 

4. 

5. 

CASE  10 

5. 

4. 

5. 

3. 

5. 

CASE  11 

5. 

4. 

4. 

2.5 

5. 

CASE  12 

4. 

3. 

3. 

, 

5. 

CASE  13 

3. 

3. 

4. 

4  . 

CASE  14 

4. 

5. 

4. 

3. 

5. 

CASE  15 

5. 

4. 

4. 

3. 

5. 

CASE  16 

5. 

5. 

5. 

4  . 

5. 

CASE  17 

4  . 

3. 

4. 

4  . 

4  . 

CASE  18 

. 

5. 

5. 

3. 

CASE  19 

3. 

3. 

4. 

5. 

4  . 

CASE  20 

5. 

4. 

4 . 

4  . 

CASE  21 

4  . 

3. 

3. 

4. 

5. 

CASE  22 

4. 

4. 

4. 

3. 

5. 

CASE  23 

5. 

3. 

3. 

2. 

3. 

CASE  24 

5. 

4. 

4  . 

4. 

4. 

CASE  25 

4. 

3. 

2. 

2. 

3. 
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COST  RISK  RELATIVE  IMPORTANCE 
MEANS,  VARIANCES,  AND  OTHER  STATISTICS 


SOURCE  SEL 

ACQ  ST PAT 

ACQ  PLAN 

CONTRACT 

No.  of  cases  : 

22 

25 

24 

24 

Average 

Standard  dev 
Skewness  : 

Kurtosis 

4.0000 

0.7559 

-0.7276 

1.1789 

3.6400 

0.8602 

0.3804 

-0.8278 

4.0000 

0.9325 

-0.3510 

-1.0455 

4.4167 

0.5836 

-0.3646 

-0.6783 

PERCENTILES  : 
0th  (Minimum) 

2.0000 

2.0000 

2.0000 

3.0000 

5th 

2.1500 

2.3000 

2.2500 

3.2500 

25th  (Lower) 

4.0000 

3.0000 

3.0000 

4.0000 

50th  (Median) 

4.0000 

3.0000 

4.0000 

4.0000 

75th  (Upper) 

4.2500 

4.0000 

5.0000 

5.0000 

95th 

5.0000 

5.0000 

5.0000 

5.0000 

100th  (Max) 

5.0000 

5.0000 

5.0000 

5.0000 

T&E 

No.  of  cases 

23 

Average 

3.0435 

Standard  dev 

0.7057 

Skewness 

-0.0611 

Kurtosis 

-0.8196 

PERCENTILES  : 

0th  (Minimum) 

2.00U0 

5th 

2.0000 

25th  (Lower) 

3.0000 

50th  (Median) 

3.0000 

75th  (Upper) 

4.0000 

95th 

4.0000 

100th  (Max) 

4.0000 
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SCHEDULE  RISK  RELATIVE  IMPORTANCE 
MEANS,  VARIANCES,  AND  OTHER  STATISTICS 


SOURCE  SEL 

ACQ  ST RAT 

ACQ  PLAN 

CONTRACT 

No.  of  cases 

22 

25 

24 

25 

Average 

3.6818 

4.0400 

4  .  uo33 

3.4400 

Standard  dev 

0.8937 

0.6758 

0.7173 

0.8699 

Skewness 

-0.1674 

-0.0465 

-0.1250 

-0.6262 

Kurtosis 

-0.5311 

-0.6058 

-0.9116 

1.5384 

PERCENTILES  : 
0th  (Minimum) 

2.0000 

3.0000 

3.0000 

1.0000 

5th 

2.0000 

3.0000 

3.0000 

1.3000 

25th  (Lower) 

3.0000 

4.0000 

4.0000 

3.0000 

50th  (Median) 

4.0000 

4.0000 

4.0000 

3.0000 

75th  (Upper) 

4.0000 

4.5000 

5.0000 

4.0000 

95th 

5.0000 

5.0000 

5.0000 

5.0000 

100th  (Max) 

5.0000 

5.0000 

5.0000 

5.0000 

T&E 

No.  of  cases 

23 

Average 

3.7391 

Standard  dev 

0.6887 

Skewness 

-0.5220 

Kurtosis 

0.8072 

PERCENTILES  : 
0th  (Minimum) 

2.0000 

5th 

2.2000 

25th  (Lower) 

3.0000 

50th  (Median) 

4.0000 

75th  (Upper) 

4.0000 

flSth 

5.0000 

100th  (Max) 

5.0000 
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TECHNICAL  RISK  RELATIVE  IMPORTANCE 
MEANS,  VARIANCES,  AND  OTHER  STATISTICS 


SOURCE  SEL 

ACQ  ST RAT 

ACQ  PLAN 

CONTRACT 

No.  of  cases 

22 

25 

24 

23 

Average 

4.4545 

3.8400 

3.7083 

3.5870 

Standard  dev 

0.5958 

0.8000 

0.9991 

0.9127 

Skewness 

-0.5526 

0.3074 

-0.7760 

0.0625 

Kurtosis 

-0.5244 

-1.3440 

0.9893 

-0.7701 

PERCENTILES  : 

0th  (Minimum) 

3.0000 

3.0000 

1.0000 

2.0000 

5th 

3.1500 

3.0000 

1.2500 

2.0000 

25th  (Lower) 

4.0000 

3.0000 

3.0000 

3.0000 

50th  (Median) 

4.5000 

4.0000 

4.0000 

4.0000 

75th  (Upper) 

5.0000 

4.5000 

4.0000 

4.0000 

95th 

5.0000 

5.0000 

5.0000 

5.0000 

100th  (Max) 

5.0000 

5.0000 

5.0000 

5.0000 

T&E 

No.  of  cases 

23 

Average 

4.5652 

Standard  dev 

0.6624 

Skewness 

-1.2882 

Kurtosis 

0.6248 

PERCENTILES  : 

0th  (Minimum) 

3.0000 

5th 

3.0000 

25th  (Lower) 

4.0000 

50th  (Median) 

5.0000 

75th  (Upper) 

5.0000 

95th 

5.0000 

100th  (Max) 

5.0000 
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STORM  DATA  SET  LISTING 
DETAILED  PROBLEM  DATA  LISTING  FOR 
PERSONAL  DATA  INDICATOR  VARIABLES 


ROW  LABEL 

DEGREE 

OPS 

EXPER 

OTHER 

BASE 

AGE >30 

MARRIED 

RISK 

TAKF*? 

CASE  1 

1. 

0. 

0. 

0. 

0. 

1 . 

CASE  2 

1. 

0. 

0. 

0. 

1 . 

0. 

CASE  3 

1 . 

0. 

0. 

0. 

1 . 

0. 

CASE  4 

1. 

0. 

0. 

0. 

0. 

0. 

CASE  5 

0. 

1. 

1 . 

1 . 

1 . 

0. 

CASE  6 

1. 

0. 

0. 

0. 

0. 

1. 

CASE  7 

0. 

1. 

1 . 

1. 

1 . 

1. 

CASE  8 

1. 

1. 

1. 

1 . 

1. 

0. 

CASE  9 

1 . 

0. 

0. 

0. 

1 . 

1. 

CASE  10 

1. 

0. 

0. 

1 . 

1. 

0. 

CASE  11 

1. 

0. 

0. 

0. 

0. 

0. 

CASE  12 

1. 

1. 

1. 

1 . 

1 . 

1 . 

CASE  13 

1 . 

0. 

0. 

1 . 

1. 

1 . 

CASE  14 

1 . 

0. 

1 . 

1. 

1 . 

1 . 

CASE  15 

1. 

0. 

1 . 

1. 

1 . 

1 . 

CASE  16 

0. 

0. 

0. 

0. 

1  . 

0. 

CASE  17 

1 . 

0. 

1. 

0. 

0. 

1 . 

CASE  18 

0. 

0. 

0. 

1 . 

1 . 

0. 

CASE  19 

1 . 

0. 

0. 

0. 

0. 

1 . 

CASE  20 

1 . 

0. 

0. 

0. 

1  . 

1 . 

CASE  21 

1 . 

0. 

0. 

0. 

0. 

1 . 

CASE  22 

1. 

0. 

0. 

0. 

1 . 

1 . 

CASE  23 

1. 

0. 

1 . 

0. 

0. 

1 . 

CASE  24 

0. 

0. 

0. 

1. 

1. 

1 . 

CASE  25 

1. 

1 . 

1 . 

1 . 

1 . 

1 . 

Note:  A  "1"  indicates  a  "yes"  response. 
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Appendix  D:  Risk  Taker  Correlation  Matrix 

RISK  DESCRIPTION  CORRELATIONS 
CORRELATION  (SAMPLE  SIZE)  MATRIX 


- DEGREE - 


- OPS  EXPER--  --OTHER  BASE-- 


DEGREE 

l.OOOC 

25) 

-0.250( 

25) 

-0.042( 

25) 

OPS  EXPER 

-0.250( 

25) 

l.OOOC 

25) 

0.667( 

25) 

OTHER  BASE 

-0.042( 

25) 

0.667( 

25) 

l.OOOC 

25) 

AGE  >  30 

-0.363( 

25) 

0.564( 

25) 

0.510( 

25) 

MARRIED 

-0.343( 

25) 

0.343( 

25) 

0. 157( 

25) 

RISK  TAKER 

0.250( 

25) 

-0.042( 

25) 

0.215( 

25) 

---AGE  > 

30--- 

- MARRIED--- 

--RISK  TAKER- 

DEGREE 

-0.363( 

25) 

-0.343( 

25) 

0.250C 

25) 

OPS  EXPER 

0.564( 

25) 

0.343( 

25) 

-0.042( 

25) 

OTHER  BASE 

0.510( 

25) 

0. 157( 

25) 

0.215C 

25) 

AGE  >  30 

l.OOOC 

25) 

0.608( 

25) 

-0.006C 

25) 

MARRIED 

0.608C 

25) 

l.OOOC 

25) 

-0. 157( 

25) 

RISK  TAKER 

-0,006C 

25) 

-0. 157( 

25) 

l.OOOC 

25) 
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Appendix  E:  Question  7  Correlation  Matrices 

COST  RISK  RELATIVE  IMPORTANCE 
CORRELATION  (SAMPLE  SIZE)  MATRIX 


--SOURCE 

SEL-- 

- ACQ  STRAT-- 

---ACQ  PLAN-- 

SOURCE  SEL 

l.OOOC 

22) 

-0. 157( 

22) 

-0. 137( 

22) 

ACQ  ST RAT 

-0. 157( 

22) 

1 . 000( 

25) 

0 .450( 

24) 

ACQ  PLAN 

-0. 137( 

22) 

0 .450( 

24) 

l.OOOC 

24) 

CONTRACT 

0. 195( 

21) 

0.269( 

24) 

0 .409( 

23) 

TiE 

0. 191( 

21) 

0.  HOC 

23) 

0.420( 

22) 

DEGREE 

-0. 160( 

22) 

-0.095( 

25) 

O.OOOC 

24) 

OPS  EXPER 

-0. 147( 

22) 

-0. 142( 

25) 

0. 112( 

24) 

OTHER  BASE 

0.000( 

22) 

-0. 174C 

25) 

0. 094( 

24) 

AGE  >  30 

0.000( 

22) 

-0.099( 

25) 

0.275( 

24) 

MARRIED 

0.2‘^S( 

22) 

-0. 191( 

25) 

O.OOOC 

24) 

---CONTRACT--- 

- T&E 

- DEGREE--- 

SOURCE  SEL 

0. 195< 

21) 

0. 191( 

21) 

-0. 160C 

22) 

ACQ  ST RAT 

Q.269( 

24) 

0.  HOC 

23) 

-0.095( 

25) 

ACQ  PLAN 

0 .409( 

23) 

0.420( 

22) 

O.OOOC 

24) 

CONTRACT 

l.OOOC 

24) 

0.401( 

22) 

-0. 165( 

24) 

T&E 

0.401( 

22) 

l.OOOC 

23) 

-0.470( 

23) 

DEGREE 

-0. 165( 

24) 

*  -0,470( 

23) 

l.OOOC 

25) 

OPS  EXPER 

0.065< 

24) 

*  0.425( 

23) 

-0. 250( 

25) 

OTHER  BASE 

0, 258( 

24) 

0. 079( 

23) 

-0.042( 

25) 

AGE  >  30 

*  0.419( 

24) 

0.326( 

23) 

-0.363( 

25) 

MARRIED 

0.206( 

24) 

0.311( 

23) 

-0.343( 

25) 

---OPS  EXPER-- 

--OTHER  BASE-- 

---AGE  > 

30-- 

SOURCE  SEL 

-0 . 147( 

22) 

o.oooc 

22) 

O.OOOC 

22) 

ACQ  ST RAT 

-0. 142( 

25) 

-0 . 174( 

25) 

-0 .099C 

25) 

ACQ  PLAN 

0. 112( 

24) 

0.094( 

24) 

0.275( 

24) 

CONTRACT 

0.065( 

24) 

0.258( 

24) 

0.419( 

24) 

T&E 

0.425( 

23) 

0.079( 

23) 

0.326< 

23) 

DEGREE 

-0. 250( 

25) 

-0.042( 

25) 

-0.363( 

25) 

OPS  EXPER 

l.OOOC 

25) 

0.667( 

25) 

0. 564( 

25) 

OTHER  BASE 

0.667( 

25} 

l.OOOC 

25) 

0. 510( 

25) 

AGE  >  30 

0.564( 

25) 

0. 510( 

25) 

1  oooc 

25) 

MARRIED 

0.343( 

25) 

0. 157( 

25) 

0.608( 

25) 
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--MARRIED-' 


SOURCE  SEL 

0.256( 

22) 

ACQ  STRAT 

-0. 191( 

25) 

ACQ  PLAN 

0.000< 

24) 

CONTRACT 

0.206( 

24) 

T&E 

0.311( 

23) 

DEGREE 

-0.343( 

25) 

OPS  EXPER 

0.343( 

25) 

OTHER  BASE 

0. 157( 

25) 

AGE  >  30 

0.608( 

25) 

MARRIED 

1.000( 

25) 

SCHEDULE 

RISK 

RELATIVE  IMPORTANCE 

CORRELATION  (SAMPLE  SIZE) 

MATRIX 

—  SOURCE 

SEL  — 

- ACQ  STRAT--  - 

--ACQ  PLAN-- 

SOURCE  SEL 

1.000( 

22) 

0. 108( 

22) 

0. 128( 

22) 

ACQ  STRAT 

0.10d( 

22) 

1 .000( 

25) 

0.695( 

24) 

ACQ  PLAN 

0. 128( 

22) 

0.695( 

24) 

1 .000( 

24) 

CONTRACT 

0.244( 

22) 

-0.244( 

25) 

0.074( 

24) 

T&E 

0. 168( 

21) 

0.098( 

23) 

-0.070( 

22) 

DEGREE 

-0. 172( 

22) 

-0.272( 

25) 

-0.231( 

24) 

OPS  EXPER 

-0.299( 

22) 

0. 121( 

25) 

0.085( 

24) 

OTHER  BASE 

-0.226( 

22) 

-0. 171( 

25) 

-0.092( 

24) 

AGE  >  30 

-0.014( 

22) 

0.068( 

25) 

0. 129( 

24) 

MARRIED 

-0.059( 

22) 

-0.218( 

25) 

-0.042( 

24) 

---CONTRACT--- 

- T&E- 

---DEGREE 

; - 

SOURCE  SEL 

0.244( 

22) 

0. 168( 

21) 

-0. 172( 

22) 

ACQ  STRAT 

-0.244( 

25) 

0.098( 

23) 

-0.272( 

25) 

ACQ  PLAN 

0.074< 

24) 

-0.070( 

22) 

-0.231( 

24) 

CONTRACT 

1.000( 

25) 

-0. 230( 

23) 

0.023( 

25) 

T&E 

-0.230( 

23) 

1.000( 

23/ 

-0.348( 

23) 

DEGREE 

0.023( 

25) 

-0.348( 

23) 

1.000( 

25) 

OPS  EXPER 

0.094( 

25) 

0.048( 

23) 

-0 . 250( 

25) 

OTHER  BASE 

0. 102( 

25) 

-0.086( 

23) 

-0.042( 

25) 

AGE  >  30 

0.015( 

25) 

0.209( 

23) 

-0.36o( 

25) 

MARRIED 

0.052( 

25) 

0.259( 

23) 

-0.343( 

25) 

---OPS  EXPER— 

—OTHER 

BASE-- 

---AGE  > 

30-- 

SOURCE  SEL 

-0.299( 

22) 

-0.226( 

22) 

-0.014( 

22) 

ACQ  STRAT 

0. 121( 

25) 

-0. 171( 

25) 

0.068( 

25) 

ACQ  PLAN 

0.085( 

24) 

-0.092( 

24) 

0. 129( 

24) 

CONTRACT 

0.094( 

25) 

0. 102( 

25) 

0.015( 

25) 

T&E 

0.048( 

23) 

-0.086( 

23) 

0.209( 

23) 

DEGREE 

-0.250( 

25) 

-0.042( 

25) 

-0 .363( 

25) 

OPS  EXPER 

1.000( 

25) 

0  667( 

25) 

0.564( 

25) 

OTHER  BASE 

0.667( 

25) 

1.000( 

25) 

0.510( 

25) 

AGE  >  30 

0.564( 

25) 

0.510( 

25) 

1 .000( 

25) 

MARRIED 

0.343( 

25) 

0. 157( 

25) 

0.608( 

25) 

- MARRIED - 

SOURCE  SEL 

-0.059( 

22) 

ACQ  STRAT 

-0.218( 

25) 

ACQ  PLAN 

-0.042( 

24) 

CONTRACT 

0.052( 

25) 

T&E 

0.259( 

23) 

DEGREE 

-0.343( 

25) 

OPS  EXPER 

0.343( 

25) 

OTHER  BASE 

0. 157( 

25) 

AGE  >  30 

0.608( 

25) 

MARRIED 

1.000( 

25) 

TECHNICAL 

RISK 

RELATIVE 

IMPORT 

ANCE 

CORRELATION  (SAMPLE  SIZE)  MATRIX 


--SOURCE  SEL--  ---ACQ  STRAT--  ---ACQ  PLAN--- 


SOURCE  SEL 

1.000( 

22) 

0.348( 

22) 

0.015( 

22) 

ACQ  STRAT 

0.348( 

22) 

l.OOOC 

25) 

0.310( 

24) 

ACQ  PLAN 

0.015( 

22) 

0.310( 

24) 

l.OOOC 

24) 

CONTRACT 

-0.202( 

20) 

0.230( 

23) 

-0.007( 

22) 

T&E 

0. 179( 

21) 

0.330( 

23) 

0.337( 

22) 

DEGREE  * 

-0.442( 

22) 

-0.357( 

25) 

-0.363( 

24) 

OPS  EXPER 

-0.051( 

22) 

-0.026( 

25) 

-0.057( 

24) 

OTHER  BASE 

-0.014( 

22) 

-0.060( 

25) 

-0.033( 

24) 

AGE  >  30 

0. 144( 

22) 

0. 181( 

25) 

0. 189( 

24) 

MARRIED 

0. 266( 

22) 

0.298( 

25) 

0.241( 

24) 
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- CONTRACT-  — 

- T&E 

- DEGREE--- 

SOURCE  SEL 

-0.202( 

20) 

0. 179( 

21) 

-0.442( 

22) 

ACQ  ST RAT 

0.230( 

23) 

0.330( 

23) 

-0.357( 

25) 

ACQ  PLAN 

-0.007( 

22) 

0.337( 

22) 

-0.363( 

24) 

CONTRACT 

1.000( 

23) 

0.263( 

21) 

-0. 126( 

23) 

T&E 

0.263( 

21) 

l.OOOC 

23) 

-0. 131( 

23) 

DEGREE 

-0. 126( 

23) 

-0. 131( 

23) 

l.OOOC 

25) 

OPS  EXPER 

0.084( 

23) 

0.028( 

23) 

-0.250( 

25) 

OTHER  BASE 

-0. 173( 

23) 

-0. 149( 

23) 

-0.042( 

25) 

AGE  >  30 

-0.085( 

23) 

0.047( 

23) 

-0.363( 

25) 

MARRIED 

-0.041( 

23) 

0.214( 

23) 

-0.343( 

25) 

---OPS  EXPER-- 

—OTHER  BASE-- 

---AGE  > 

30-- 

SOURCE  SEL 

-0.051( 

22) 

-0.014( 

22) 

0. 144( 

22) 

ACQ  STRAT 

-0.026( 

25) 

-0.060( 

25) 

0. 181( 

25) 

ACQ  PLAN 

-0. 057( 

24) 

-0.033( 

24) 

0. 189( 

24) 

CONTRACT 

0.084( 

23) 

-0. 173( 

23) 

-0.085( 

23) 

T&E 

0.028( 

23) 

-0. 149( 

23) 

0.047( 

23) 

DEGREE 

-0.250( 

25) 

-0.042( 

25) 

-0.363( 

25) 

OPS  EXPER 

1.000( 

25) 

0.667( 

25) 

0.564( 

25) 

OTHER  BASE 

0.667( 

25) 

1.000( 

25) 

0.510( 

25) 

AGE  >  30 

0.564( 

25) 

0.510( 

25) 

l.OOOC 

25) 

HARRIED 

0.343( 

25) 

0. 157( 

25) 

0.608C 

25) 

- MARRIED--- 

SOURCE  SEL 

0.266( 

22) 

ACQ  STRAT 

0.298( 

25) 

ACQ  PLAN 

0.241( 

24) 

CONTRACT 

-0.041( 

23) 

T&E 

0.214( 

23) 

DEGREE 

-0.343( 

25) 

OPS  EXPER 

0.343( 

25) 

OTHER  BASE 

0. 157( 

25) 

AGE  >  30 

0.608( 

25) 

MARRIED 

1 .000( 

25) 
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Appendix  F:  Question  9  Data 

STORM  DATA  SET  LISTING 
DETAILED  PROBLEM  DATA  LISTING  FOR 
RISK  ANALYSIS/ASSESSMENT  THOROUGHNESS 


ROW  LABEL 

XI 

CASE 

1 

50. 

CASE 

2 

65. 

CASE 

3 

60. 

CASE 

4 

40. 

CASE 

5 

70. 

CASE 

6 

70. 

CASE 

7 

50. 

CASE 

8 

50. 

CASE 

9 

60. 

CASE 

10 

60. 

CASE 

11 

35. 

CASE 

12 

50. 

CASE 

13 

85. 

CASE 

14 

35. 

CASE 

15 

65. 

CASE 

16 

30. 

CASE 

17 

40. 

CASE 

18 

50. 

CASE 

19 

35. 

CASE 

20 

40. 

CASE 

21 

50. 

CASE 

22 

55. 

CASE 

23 

10. 

CASE 

24 

45. 

CASE 

25 

70 . 
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RISK  ANALYSIS/ASSESSMENT  THOROUGHNESS 
MEANS,  VARIANCES,  AND  OTHER  STATISTICS 


XI 

No.  of  cases  selected 

25 

Average 

50.8000 

Standard  deviation 

16.0520 

Skewness 

-0.2557 

Kurtosis 

0.6691 

PERCENTILES  : 

0th  (Minimum) 

10.0000 

5th 

16.0000 

25th  (Lower  Quartile) 

40.0000 

50th  (Median) 

50.0000 

75th  (Upper  Quartile) 

62.5000 

95th 

80.5000 

100th  (Maximum) 

85.0000 
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This  preliminary  study  took  responses  from  a  series  of 
interviews  with  program  managers  to  assess  the  program 
managers'  knowledge  of  risk  and  risk  assessments,  how  they 
usually  performed  risk  assessments,  why  they  performed  risk 
assessments,  and  if  any  relationships  existed  between 
individual  backgrounds  and  risk  taking  behaviors. 

The  research  found  that  although  the  program  managers 
in  this  study  were  all  involved  with  risk  assessments  and 
analyses,  they  did  not  appear  to  be  properly  educated  in  the 
areas  of  risk  and  risk  assessment.  The  managers  were, 
however,  interested  in  learning  more  about  the  concept  of 
risk  and  strongly  suggested  the  addition  of  formal 
coursework  in  the  area  of  risk  management. 

This  study  also  showed  that  differences  are  likely  to 
exist  between  program  managers  possessing  technical  degrees 
(engineering,  basic  sciences,  etc.)  and  those  not  possessing 
technical  degrees  with  respect  to  how  cost  and  technical 
i isk  arc  evaluated  in  various  program  phases.  To  get  a  more 
in-depth  understanding  of  the  relationship  of  risk  to  Air 
Force  weapon  systems  acquisition,  it  is  important  to 
understand  why  these  differences  exist. 

Consequently,  it  is  recommended  that  further  research 
be  carried  out  to  more  precisely  measure  this  difference  in 
risk  taking  tendencies. 


